whatslab

English | 한국어

whatslab is the WHATs LAB teleoperation core — a pure-Python SDK that turns human motion (VR controllers, hand-tracking, data gloves) into robot arm and hand joint angles. It is developed at WHATs LAB as the shared logic layer beneath our simulators (MuJoCo, Isaac Sim) and ROS2 stack.

whatslab has no dependency on ROS and runs in-process anywhere. It provides the parts — input receivers, calibration, hand/arm retargeting, visualization, dataset recording — and leaves the assembly (wiring a pipeline into a simulator or robot) to the consumer. Inputs are normalized to a single canonical frame (x=forward, z=up, right-handed), so downstream code never re-maps axes.

Table of contents

• Main features
• Installation
• Compatibility
• Quick start
• Examples & tools
• Documentation
• Acknowledgments
• License

Main features

whatslab is framework-agnostic:

• pure Python, zero ROS dependency — used in-process from MuJoCo, Isaac Sim, or ROS2
• provides composable parts; the consumer owns the pipeline
• strict dependency direction (receiver → core, model → core·robot)

whatslab is retargeting-first:

• hand: human-hand URDF joint angles from the glove → one forward pass of a single learned ONNX model. No per-frame IK — 900Hz+ on one CPU thread, every robot hand and both sides share one graph
• arm: pinocchio analytic Jacobian + damped least squares
• output is {side: {joint_name: rad}}, ready to publish

whatslab is calibrated:

• wrist yaw alignment (head-relative snapshot)
• per-user arm reach scaling, persisted into the rig config

Supported hands

One ONNX graph covers every hand below, both sides. The name in the first column is what a rig's retarget: field takes; the alias in parentheses also resolves.

Hand	retarget:	URDF in dexhand-description
Human reference hand (retargeting input)	human (base_hand)	base_hand/urdf/{side}.urdf
ORCA Hand	orca (orca_hand)	orca_hand/urdf/{side}.urdf
Allegro Hand	allegro (allegro_hand)	allegro_hand/allegro_hand_{side}.urdf
Tesollo DG-5F	tesollo (tesollo_dg5f)	tesollo_dg5f/dg5f_{side}.urdf
ROBOTIS HX5-D20	robotis (robotis_hx5_d20)	robotis_hx5_d20/urdf/hx5_d20_{side}.urdf

Any other hand raises on construction, listing the names the table does hold. To add one, generate its tables with retarget_net's tools/onboard_urdf.py — the URDF must carry the sensor-frame contract ({side}_sensor_dorsum plus _proximal/_distal per finger). A hand the model was not trained on needs a few hundred fine-tuning steps and a graph re-export before it performs.

Installation

Publicly available under a source-available license (not published on PyPI — install from source).

pip install '.[all]'      # receiver + hand + arm + viz + data
pip install '.[hand]'     # partial: hand / arm / receiver / viz / data
pip install -e '.[all]'   # editable, for development

Teleoperating real hardware (nero arm + ORCA hand) needs extra drivers. scripts/install_robot_deps.sh handles them in one step.

PY=$(which python) ./scripts/install_robot_deps.sh
python examples/quest_arm.py --rig rigs/nero_orca_right.yaml --viz --robot

pyAgxArm (CAN) ships in the robot extra, but orca_core cannot: every tag declares numpy>=2.2.6 while this repo pins numpy<2, so resolution fails (the code itself runs fine on numpy 1.26). Its wheel also omits the hand configs (models/<version>/<model>/config.yaml), so the script clones the source and installs it editable. The hand model defaults to whatever orca_core picks; pass --hand-model for a different one.

Robot/rig configs are bundled. URDF and meshes are provided by the separate single-source package dexhand-description, pulled in by the hand/arm extras; override the asset tree with WHATSLAB_MODELS_ROOT.

Compatibility

Data-glove teleoperation goes through Spine, WHATs LAB's glove middleware. Supported Spine versions: 2.3.1 and below (newer versions are not yet supported). Controller / hand-tracking (Quest) paths do not require Spine.

The glove path follows Spine's OSC contract as documented in Spine's docs/OSC_Protocol.md: GloveHumanHandReceiver consumes /{side}/quat/get, and GloveRobotHandReceiver consumes /{side}/joint_angles/get (name/angle pairs) plus /{side}/wrist/get. Every Spine message carries the message type in args[0].

Quick start

from whatslab.teleop import GloveModel

m = GloveModel("rigs/nero_orca_right.yaml")   # arm = controller IK, hand = glove retarget
m.start()

while True:
    q = m.get_q()             # {"right": {joint_name: rad, ...}}  — arm + hand merged
    publish_joint_states(q)   # consumer's job: reorder into sim/ROS joint order

Presets: QuestModel (hand-tracking), GloveModel (controller + glove), HandModel (hand only). GloveModel takes its arm target from either transport — GloveModel(rig, arm_source="quest") (OSC/UDP, default) or arm_source="webxr" (browser WebXR over WebSocket, no APK to sideload). The WebXR path defaults to wireless: it serves HTTPS with a self-signed cert on this machine's LAN IP, since WebXR needs a secure context. Pass tls=False for the wired adb reverse route. See docs/API.md. Both transports can be installed side by side. For a custom hardware combination, subclass TeleopModel and implement the single abstract hook _get_raw_target() — it decides which source feeds the arm EE target. Everything else (calibration, IK, retargeting, safety) is already wired.

Examples & tools

python examples/quest_arm.py --rig rigs/nero_orca_right.yaml            # controller + glove
python examples/quest_arm.py --rig rigs/nero_orca_right.yaml --arm wrist  # Quest hand-tracking
python examples/verify_rig.py --rig rigs/nero_orca_right.yaml           # inspect rig kinematics

python tools/align_frames.py robot --robot robots/nero.yaml            # align a robot to canonical axes

Judge arm-IK accuracy on targets generated by FK from valid joint angles — the error floor is then exactly 0, so whatever error remains is the solver's. Coordinate-space trajectories pass through unreachable poses and conflate solver quality with reachability. (The fixed arm-IK benchmark is internal tooling and is not shipped here.)

Run the test suite with pip install -e '.[all,dev]' && pytest.

Documentation

• Real-robot quickstart (Korean) — clone to running teleop on a nero arm + ORCA hand, and what usually goes wrong.
• Guide — bringing up a new robot, calibration workflow, arm-IK tuning and how to judge a change, diagnostics, sending to real hardware.
• API reference — public symbols per subpackage, with signatures.
• Changelog — version history. 0.2.0 contains breaking changes (model.ik[s] → model.sides[s].ik, RobotModel.solve removed).

Acknowledgments

whatslab builds on excellent open-source work: Pinocchio (rigid-body kinematics/IK), viser (web 3D visualization), and LeRobot (dataset format).

License

Licensed under the 주식회사 왓츠랩 (WHATs LAB Corp) Source Code License (based on CC BY-NC-ND 4.0) — source-available, non-commercial, no derivatives. See LICENSE.

Copyright © 주식회사 왓츠랩 (WHATs LAB Corp). All rights reserved.