Repository navigation
Conversation
Adds `jmex`, a plugin for the j-mex AgileMaster motion capture suit, and an example, in two parts, that consumes what it publishes. The plugin reads one MOXI Player channel through the MOXI Receiver SDK (0.8.0 or newer, obtained from j-mex; without it the plugin is skipped and the rest of the tree builds as before). It fills the 24 `FullBodyPose` slots from the standard human skeleton Player has loaded, one slot per joint, already in OpenXR axes, and pushes them under the `body.jmex` vendor on every frame. A slot with no joint, or with no usable value in a frame, goes out invalid; nothing is inferred from its neighbors. The raw device clock carries the time Player stamped on the frame (`--device-clock=local` uses the pickup time instead), and if the CloudXR runtime restarts underneath the plugin, the plugin reconnects rather than ending the run. Isaac Teleop learns `body.jmex` as a full-body vendor: a live tracker that reads the plugin's collection, with `collection_id` and `max_flatbuffer_size` parameters, its entry in the DeviceIO factory, the vendor validation cases, and a row in `devices.yaml`. An SDK unpacked into the plugin directory is kept out of the clang-format check. `isaaccapture_examples.jmex.jmex_g1_tasks` registers a task that inherits Isaac Lab's `LocomanipulationG1EnvCfg` and takes its action from the operator; nothing in Isaac Lab is modified. It is upper-body teleop. Each arm is rebuilt segment by segment onto the robot's own shoulder, so the operator's arm angles survive while the lengths stay the robot's, and the elbows and shoulders get IK targets of their own from that same chain. The wrist's orientation is the operator's hand, carried across through a forearm frame. The operator's torso turns the robot's shoulders about its waist, so leaning, bending sideways and twisting come through; their heading does not, so the operator can turn to face a screen without moving the robot. Only a frame that carries all 24 slots drives the robot, and slots that stay missing are named on the console. `isaaccapture_examples.jmex.record_jmex_full_body` writes the same stream to a standard full-body MCAP. Tests cover the plugin's slot mapping, its change of axes, the device clock choice and the session rebuild after a failed push, the vendor's parameters, and the example's action source, including the geometry of the rebuilt arm and the orientation carried through the forearm frame. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: j-mex <jmex.dev@jmex.com.tw>
Contributor
|
📝 Docs preview is not auto-deployed for fork PRs. A maintainer with write access to |
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 50.20% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 249 functions across 23 files. (18 skippe… | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly summarizes the main changes: adding the AgileMaster full-body plugin and the G1 teleoperation example. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 50.20% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 249 functions across 23 files. (18 skipped: 18 unsupported.)
- Fix all pre-merge checks with AI
✨ Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Contributor
|
/preview-docs |
Contributor
|
✅ Preview deployed: https://NVIDIA.github.io/IsaacCapture/preview/pr-1250/ |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds support for the j-mex AgileMaster motion capture suit: a device plugin that publishes the
operator's body pose as
FullBodyPoseunder a newbody.jmexvendor, the core registration thatlets Isaac Teleop read it, and an Isaac Lab example that drives a Unitree G1 from it. None of this
is upstream yet; the plugin, the example and their tests are all new.
AgileMaster has three parts: the Mocap Suit the operator wears; MOXI Player, which solves the suit
onto a skeleton and streams it on a numbered channel; and the MOXI Receiver SDK, which the plugin
links against.
The plugin (
src/plugins/jmex/) receives one Player channel and fills the 24BodyJointslotsfrom the standard human skeleton loaded in Player, one slot per joint, in OpenXR axes. It pushes one
FullBodyPoseper frame, on thejmex_full_bodycollection by default. A slot with no joint, orwith no usable value in a frame, goes out with
is_validfalse; nothing is inferred fromneighboring joints. The raw device clock carries the time Player stamped on the frame
(
--device-clock=localuses the pickup time instead). If the CloudXR runtime restarts underneathit, the plugin re-establishes its session and carries on. It needs MOXI Receiver SDK 0.8.0 or newer;
without it, configure prints
Skipping jmex plugin build: MOXIReceiverSDK 0.8.0 or newer not foundand the rest of the tree builds as before.
Where the conversion happens. Player streams its own skeleton. The plugin maps it onto the 24
slots and into OpenXR axes (
body_joint_mapping,openxr_convention); Player does not emitFullBodyPose. There are two reasons for this. The change of axes was measured in this tree and ispinned by the plugin's tests, next to the code that consumes the slots. Done in Player, those tests
would live where this repository's CI never runs them. And Player does not have to produce Isaac
Teleop's internal schema, so a change to
FullBodyPosestays in this tree instead of waiting on aPlayer release.
Registration in core. Outside the new directories, each change adds an entry to an existing
list; no existing line is changed or removed.
src/core/live_trackers/cpp/live_full_body_tracker_jmex_impl.{hpp,cpp}body.jmex. It reads the plugin's collection and takes two vendor parameters,collection_idandmax_flatbuffer_sizesrc/core/live_trackers/cpp/CMakeLists.txtlive_deviceio_factory.{hpp,cpp}try_create_*function; one row ink_tracker_dispatch, after the existing full-body vendors, soFullBodyTracker's default vendor is unchanged; one branch invalidate_vendor_selectionstests/cpp/core/live_trackers/test_vendor_validation.cppSECTIONs forbody.jmex, beside the existing vendors'src/plugins/CMakeLists.txtadd_subdirectory(jmex)in the ungated list; the plugin skips itself without the SDKtests/cpp/plugins/CMakeLists.txt,tests/python/examples/CMakeLists.txtadd_subdirectory(jmex)for the plugin's and the example's tests, with a one-line note in the firstcmake/ClangFormat.cmakedocs/source/_data/devices.yamlThe example (
examples/jmex/, packageisaaccapture_examples.jmex) has two parts:jmex_g1_tasksregistersIsaac-PickPlace-Locomanipulation-G1-Jmex-Abs-v0, which inherits IsaacLab's
LocomanipulationG1EnvCfgand takes its action from the operator; nothing in Isaac Lab ismodified. It is upper-body teleop. Each arm is rebuilt segment by segment onto the G1's own
shoulder, so the operator's arm angles carry over at the robot's arm lengths, and the elbows and
shoulders get IK targets from that same chain. The wrist's orientation is the operator's hand,
carried through a forearm frame. The operator's torso turns the robot's shoulders about its waist,
so leaning, bending sideways and twisting come through; their heading does not, so the operator
can turn to face a screen without moving the robot. Only a frame that carries all 24 slots drives
the robot.
record_jmex_full_bodyrecords the same stream to a standard full-body MCAP(
core.FullBodyPoseRecordon thefull_bodychannel), which the sharedreplay_full_bodyscriptplays back unmodified.
Type of change
Testing
The tests in this PR need no SDK, so CI builds and runs them.
ctest -L jmexselects 15: theplugin's 14 Catch2 cases (slot mapping, change of axes, device clock choice, one frame becoming one
FullBodyPose, session recovery after a failed push), which compile the plugin's SDK-free sourcesdirectly; and the example's Python test, whose 91 pytest cases cover its action source, including
the geometry of the rebuilt arm and the orientation carried through the forearm frame. The Python
test is registered when
BUILD_EXAMPLESandBUILD_VIZare on, as in the ubuntu and Windows jobs.The
body.jmexsections of the vendor validation test run underctest -R 'vendor validation'.Local gate, Ubuntu x86_64. We ran the six x64 entries of
build-ubuntu(Python 3.11, 3.12 and3.13, each in Debug and Release), configured as that job does and without the SDK, each from an empty
tree, twice: once with CI's toolchain (GCC 11.4, CMake 3.31.6, uv 0.10.6, clang-format 14.0.0) and
once with GCC 13.3. In all twelve:
jmextests and the 3 vendor validation cases passed;.cppfiles that build without the SDK compiled without warnings (theother two are covered below);
ctest, 495 passed, 9 skipped and 3 failed. The three are upstreamGPU tests (
VkContextwith an explicit device index, and twocamera_viztests) that fail on thisworkstation's GPU setup and are unrelated to this change; CI's runners have no GPU and skip them.
The two plugin sources that compile only with the SDK were built against SDK 0.8.0 with GCC 13 and
syntax-checked with GCC 11, without warnings. With CI's toolchain we also repeated the Release
install with
auditwheel repair, thepip-editable-installjob (its import check, and itswheel-content tests: 6 passed) and the configure of
test-viz-sanitizers.SKIP=check-copyright-year pre-commit run --all-files,make current-docs(-W),scripts/check_doc_refs.py --all, ruff, and clang-format 14 on the changed files all pass. Windowsand arm64 were not built locally.
Since this PR comes from a fork, its workflows wait for a maintainer's approval. The same commit has
already been through this repository's CI on our fork, where all twelve
build-ubuntuentries (x64and arm64) passed, along with DCO, pre-commit and the docs build:
https://github.com/jmex-sw/IsaacTeleop/actions/runs/38352128694
With the device. Driven live by an operator in the suit (Isaac Lab 3.0 on Isaac Sim 6.0.1, XR
running), the G1's waist followed lean, side bend and twist with a correlation of 0.99, 90–270 ms
behind, and turning on the spot through ±90° moved its yaw less than 12°. Replayed headless from a
recording, the waist followed all three with a correlation of 1.00, one or two physics steps behind.
Running it yourself. Isaac Lab needs this repository's
isaaccaptureinstalled in its Python:no released package carries the
body.jmexvendor yet, and Isaac Lab pins the package under itsprevious name. The example README has the steps. The SDK and MOXI Player are not distributed with
this PR. A maintainer validation package can be provided through an agreed private channel on
request.
Checklist
SKIP=check-copyright-year pre-commit run --all-filesgit commit -s) per the DCOSummary by CodeRabbit