Hand tracking

The Varjo OpenXR runtime supports the XR_EXT_hand_tracking extension, which allows developers to access hand tracking data with Varjo XR-3 and VR-3 headsets using Unreal’s built-in OpenXRHandTracking plugin.

Using hand tracking with Unreal

Make sure that the OpenXRHandTracking plugin is enabled in the project. It should be enabled automatically if the Varjo OpenXR plugin is also enabled in your project.

When hand tracking data is available, you can access it with the Get Motion Controller Data blueprint node. Hand key data for each hand is provided in the Hand Key Positions, Hand Key Rotations, and Hand Key Radii arrays.

Hand tracking visualization

This example shows a simple way to visualize the tracked hands. Start by adding an empty actor in your scene and converting it into a blueprint.

Open the created blueprint.

We will visualize the hands by rendering a cube for each hand key. First, add an Instanced Static Mesh component. Select a static mesh and assign a material. Make sure to set Transform > Mobility to Movable.

Open the Event Graph of the blueprint.

Clear any existing instances of the Instanced Static Mesh and get the latest hand tracking data with Get Motion Controller Data in the Tick event.

Use the Break XRMotionControllerData node to access individual variables in the struct. Continue only if the provided Motion Controller Data is valid.

Use the length of Hand Key Positions to iterate through the arrays using a for loop. Use Get nodes to get Hand Key Position, Hand Key Rotation, and Hand Key Radius for each index.

In the body of the for loop, use the Make Transform node to create a Transform for each Key. Then, use Add Instance World Space to create an instance of the Instanced Static Mesh with the given Transform.

Repeat the steps for the other hand.

If you now run the scene in VR Preview, you should see the tracked hands being rendered. If you didn’t disable collision for the Instanced Static Mesh, you can also interact with physics.

You can further fine-tune the visualization, for example by hiding the hand when it’s near a VR controller.

Disable motion controllers being rendered in hand location

Hand key data is currently provided inside the Motion Controller Data struct. As a side effect, if you have Motion Controller components with Visualization enabled in your scene but have no motion controllers connected, the Motion Controller component renders the controller models using the location of the user’s hands.

To disable this behavior, you can hide the motion controllers (for example in the VRPawn blueprint included with the VR Template) when the Tracking Status in the Motion Controller Data is Not Tracked.