Mixed Reality

Mixed reality support

Enable mixed reality

For Varjo XR-3, you can enable mixed reality by enabling blend mode. This can be done with the XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND flag. Detailed specifications can be found in the OpenXR specifications.

XrFrameEndInfo frameEndInfo{XR_TYPE_FRAME_END_INFO};         
frameEndInfo.displayTime = frameState.predictedDisplayTime;         
frameEndInfo.environmentBlendMode = mrEnabled ? XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND : XR_ENVIRONMENT_BLEND_MODE_OPAQUE;        
frameEndInfo.layerCount = 1;         
frameEndInfo.layers = layers;  

Toggle depth estimation

Once the mixed reality cameras are enabled, Varjo runtime can estimate depth for incoming video frames. A typical use case is hand occlusion. To use depth estimation, enable the XR_VARJO_environment_depth_estimation extension and call the following function:

xrSetEnvironmentDepthEstimationVARJO(session, XR_TRUE);

Depth estimation requires that the application submit a depth buffer as well. More information about this extension is available in the OpenXR specification.

Limit depth testing range

In most cases, mixing real-world and VR content should happen in a limited range. For example, an application may want to show hands on top of VR content but does not want to show the surrounding walls. In this case XR_VARJO_composition_layer_depth_test can be used to enable depth testing only at a certain range.

The following example is taken from the specification and enables depth testing at a distance of 0 to 1 meter:

XrCompositionLayerProjection layer{XR_TYPE_COMPOSITION_LAYER_PROJECTION};
layer.space = ...;
layer.viewCount = ...;
layer.views = ...;
layer.layerFlags = ...;
XrCompositionLayerDepthTestVARJO depthTest{XR_TYPE_COMPOSITION_LAYER_DEPTH_TEST_VARJO, layer.next};
depthTest.depthTestRangeNearZ = 0.0f; // in meters
depthTest.depthTestRangeFarZ = 1.0f; // in meters
layer.next = &depthTest

In this example, between 0 and 1 meter the compositor will take depth into account from both the video layer and the application layer. From 1 meter on, it will ignore depth completely and VR content will overwrite pixels from the video stream.

Marker tracking

The Varjo XR-3 supports marker tracking. The Varjo OpenXR runtime fully implements the XR_VARJO_marker_tracking extension (since Varjo Base 3.3). See the official specification for details.