Mixed Reality

Mixed reality support

Enable mixed reality

You can enable mixed reality by enabling blend mode. This can be done with the XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND flag. Detailed specifications can be found in the OpenXR specifications.

XrFrameEndInfo frameEndInfo{XR_TYPE_FRAME_END_INFO};         
frameEndInfo.displayTime = frameState.predictedDisplayTime;         
frameEndInfo.environmentBlendMode = mrEnabled ? XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND : XR_ENVIRONMENT_BLEND_MODE_OPAQUE;        
frameEndInfo.layerCount = 1;         
frameEndInfo.layers = layers;  

Toggle depth estimation

Once the mixed reality cameras are enabled, Varjo runtime can estimate depth for incoming video frames. A typical use case is hand occlusion. To use depth estimation, enable the XR_VARJO_environment_depth_estimation extension and call the following function:

xrSetEnvironmentDepthEstimationVARJO(session, XR_TRUE);

Depth estimation requires that the application submit a depth buffer as well. More information about this extension is available in the OpenXR specification.

Limit depth testing range

In most cases, mixing real-world and VR content should happen in a limited range. For example, an application may want to show hands on top of VR content but does not want to show the surrounding walls. In this case XR_VARJO_composition_layer_depth_test can be used to enable depth testing only at a certain range.

The following example is taken from the specification and enables depth testing at a distance of 0 to 1 meter:

XrCompositionLayerProjection layer{XR_TYPE_COMPOSITION_LAYER_PROJECTION};
layer.space = ...;
layer.viewCount = ...;
layer.views = ...;
layer.layerFlags = ...;
XrCompositionLayerDepthTestVARJO depthTest{XR_TYPE_COMPOSITION_LAYER_DEPTH_TEST_VARJO, layer.next};
depthTest.depthTestRangeNearZ = 0.0f; // in meters
depthTest.depthTestRangeFarZ = 1.0f; // in meters
layer.next = &depthTest

In this example, between 0 and 1 meter the compositor will take depth into account from both the video layer and the application layer. From 1 meter on, it will ignore depth completely and VR content will overwrite pixels from the video stream.

Marker tracking

The Varjo headsets supports marker tracking. The Varjo OpenXR runtime fully implements the XR_VARJO_marker_tracking extension (since Varjo Base 3.3). See the official specification for details.