Dexterous IN-HAND ROTATION VIA Joint-wise NEURAL DYNAMICS Model
DexNDM presents a sim-to-real approach that enables unprecedented in-hand rotation in the real world. We master a wide object distribution, including challenging geometries and complex shapes, across rich wrist orientations.
16cm x 3cm x 3cm (2x)
20cm x 3cm x 3cm (2x)
14cm x 3cm x 3cm (2x)
20cm x 4cm x 4cm (2x)
16cm x 3cm x 3cm (2x)
16cm x 3 cm x 23cm (2x)
Complex Geometry (2x)
Small Objects (2x)
-z Axis. Reorienting diverse objects in the air along the principle axis (0, 0, -1).
-z Axis. Reorienting challenging small objects in the air along the principle axis (0, 0, -1).
z Axis. Reorienting diverse objects in the air along the principle axis (0, 0, 1).
-y Axis. Reorienting diverse objects in the air along the principle axis (0, -1, 0).
x Axis. Reorienting diverse objects in the air along the principle axis (1, 0, 0).
y Axis. Reorienting diverse objects in the air along the principle axis (0, 1, 0).
-x Axis. Reorienting diverse objects in the air along the principle axis (-1, 0, 0).
Various Rotation Axes. Reorienting objects in the air across diverse rotation axes, beyond principle x, -x, y, -y, z, -z.
Diverse Wrist Orientations. Reorienting objects in the air under rich wrist orientations, including palm up, palm down, thumb up, thumb down, base up, base down, and more general orientations. We also demonstrate the gravity-invariant rotation under changing gravity axes.
Rich wrist orientations along with diverse rotation axes. Reorienting objects in the air across diverse rotation axes under various wrist orientations.
Comparisons to AnyRotate. We show superior performance in rotating challenging objects used in AnyRotate. Objects are either 3D printed or bought in the store.
Comparisons to Visual Dexterity. We show on-par or superior performance in rotating challenging objects used in Visual Dexterity. Objects are either 3D printed.
Comparisons to UAN and ASAP. They cannot work in our task, failing in rotating the most basic cylinder shape.
In-Hand Translations. Effectiveness of DexNDM on in-hand translation tasks (added during rebuttal).
Out-of-Distribution Behaviour (added during rebuttal).
For teleoperation demos, all low-level actions of the robotic hand are executed entirely by the learned policy, conditioned only on high-level rotation-axis commands specified by the human operator (via pressing buttons in the controller of Quest 3 or by adjusting the controller's pose -- please refer to Sec. C in the supplement text for more details). The robotic arm is teleoperated directly by the human arm.
Assembly Tasks. Teleoperating the robot to complete assembly tasks: lightbulb (4x), four-legged table (4x), plug (2x), four-legged short table (4x), round table (style 1, 4x), round table (style 2, 4x), nut (2x), nut v2 (2x), lid (4x), lightbulb horizontal (2x), round table horizontal (2x).
Tool-Using Tasks. Teleoperating the robot to use tools: screwdriver (4x), springe (4x), thin pen (4x), pen (4x), hammer-90 (2x), brush (2x), hammer-270 (2x), cut banana using knife (2x), spoon (2x), flat brush (2x).
Orientation-Sensative Placement Tasks. Teleoperating the robot to grasp-reorient-place objects: small cup (2x), corn (2x), broccoli (2x), bread (2x), rice dumpling (2x), sweet potato (2x), steamed bun (2x), pen (2x), thin pen (2x).