Hand Tracking

Hand tracking is a technology that allows mixed reality (MR) systems to accurately detect and track the user's hand movements and finger positions in real time. By using depth sensors and computer vision algorithms, MR devices can identify the position, orientation, and articulation of each hand.

With hand tracking, users interact with virtual environments using natural gestures, such as pointing, pinching, or grabbing, mirroring how they would manipulate objects in the real world. This allows for a more intuitive and immersive user experience, reducing the learning curve and increasing the sense of presence.

The system continuously captures and interprets hand poses and movements in real time, providing a detailed model of the user's hand. Hand tracking supports a wide range of use cases, including user interface navigation, direct manipulation of virtual objects, and execution of gesture-based commands.

UNIGINE provides built-in, native support for hand tracking based on the OpenXR Standard. This allows VR applications created with UNIGINE to track user's hands and fingers in real time. The native hand tracking system supports the following OpenXR Extensions:

• XR_EXT_hand_tracking - core extension for tracking hand joints and providing their positions and orientations.
• XR_EXT_hand_interaction - enhances the basic tracking by providing interaction-specific hand data.
• XR_EXT_hand_joints_motion_range - offers additional control over the reported range of motion for joints, useful for improved realism and precision.
• XR_EXT_hand_tracking_data_source - provides information about the source or method used for hand tracking data.

This feature is integrated into the engine's VR system (no extra plugin needed) and supports any OpenXR - compatible device that offers hand tracking capabilities.

When a UNIGINE VR-based application with OpenXR backend starts, the engine checks for an available OpenXR runtime provided by the user's VR device or a platform. Then the engine automatically initializes native hand tracking and retrieves all the hand related data via the OpenXR API.

To use hand tracking, make sure to launch the engine with the OpenXR backend:

-vr_app openxr

Native hand tracking is only available in this mode. Other VR backends (such as OpenVR or Varjo Native SDK) do not support it and require external plugins to function. In this cases, hand tracking can still be enabled through the LeapMotion or Ultraleap plugins, but only if the necessary hardware is connected.

Note that some Varjo headsets (Varjo XR-3, VR-3, VR-2 Pro) include built-in Ultraleap sensors, but to access them, either the OpenXR runtime must be used or the Ultraleap plugin must be manually enabled in the application.

OpenXRExt::init(): Extension "XR_EXT_hand_tracking" found
OpenXRExt::init(): Extension "XR_EXT_hand_interaction" found
OpenXRExt::init(): Extension "XR_EXT_hand_joints_motion_range" found
OpenXRExt::init(): Extension "XR_EXT_hand_tracking_data_source" found

All hand tracking data is spatially relative to the main camera (i.e. the player's HMD - head-mounted display). This means that hand positions follow the headset, when the user moves or rotates their head, the hands move accordingly in the same local tracking space. They are not positioned in world space independently of the HMD.

Each hand's skeletal model consists of 26 bones organized in a hierarchy under a single root. The bones include the Wrist, Palm, and Finger bones for all five fingers (thumb, index, middle, ring, little). The finger bones, except the tips, are named after the corresponding bone at the further end of the bone from the fingertips.

The Wrist is considered as the root of the hierarchy for the hand - it's the point where the hand would connect to the forearm. Under the Wrist, all finger bones are arranged.

The Palm bone is located at the center of the middle finger's metacarpal bone, representing a convenient central point (the middle of the palm area), which can be useful for certain calculations or attaching objects to the user's hand.

Unlike the other fingers, the thumb lacks an intermediate phalanx, resulting in one less bone - just like in a real human hand.

All the hand tracking data is given in UNIGINE’s native coordinate system, which is a right-handed Cartesian coordinate system: X and Y form a horizontal plane, Z axis points up. When looking at a hand from the back (palm facing downward), the axes of each bone are aligned as follows:

• X-axis points to the left
• Y-axis points backwards, along the wrist
• Z-axis points upward, out of the back of the hand

This consistent basis orientation applies to all bones in the hand.

Hand tracking feature is accessible through a set of specialized classes in the engine's VR API. These classes provide a structured way to get hand and finger bone information, and to control debug visualization. The main classes involved are:

• VRHandTracking - The manager class for the hand tracking system. Through VRHandTracking you can access the two tracked hands and configure global settings related to hand tracking visualization.

• VRHand - This class represents a single hand (either left or right) being tracked. Each VRHand contains the set of bones (as VRBone objects) for that hand. The class also provides access to state information such as whether the hand is holding a controller or whether the controller is active. It also includes methods for querying individual bones and rendering a debug visualizer.

• VRBone - This class represents a single bone in the hand’s skeletal model. Each VRBone provides the position and orientation of itself and additional information like its radius (approximate thickness) and velocity. It also includes a method for rendering a debug visualizer for the specific bone.

The system updates the VRBone transforms every frame in sync with the VR loop, so at any time you can query these classes to get the latest hand pose.

The following console commands control the hand tracking debug visualizer. They can be entered into the Console (or set in configuration files or called via the API).

With the introduction of native OpenXR hand tracking, UNIGINE supports a unified and preffered approach for hand and finger tracking. However, the LeapMotion and the Ultraleap plugins are available and functional for cases where OpenXR is not an option or specific hardware configurations (such as Varjo Native SDK or OpenVR) require them.

LeapMotion Plugin (Legacy)#

Ultraleap Plugin (for Specific Use Cases)#