Mixed Reality with Unreal Engine 4

Developing mixed reality applications for XR-3 is quick and easy with Varjo OpenXR plugin. Follow these instructions to get started.

Once you have set up the Varjo OpenXR plugin (see Getting Started with Unreal) there are a couple of additional steps to complete. Navigate to Varjo OpenXR plugin settings in Project Settings > Plugins > Varjo OpenXR. To create a mixed reality application, select Alpha Blend as the Preferred Environment Blend Mode.

When Alpha Blend is selected, the video pass-through cameras are enabled when the game is started. Ability to toggle mixed reality on and off during runtime is coming later. See Unreal OpenXR roadmap for more information.

When Alpha Blend mode is enabled, you will see the image from the video pass-through cameras when your application renders 0 in the color buffer.

The easiest way to control the alpha channel is by using custom post processing. First you need to enable the alpha channel. In Project Settings > Rendering > Postprocessing, set Enable alpha channel support in post processing to Allow through tonemapper.

Add a Post Process Volume in your scene and enable Infinite Extent (Unbound) under Post Process Volume Settings in the Details panel.

Next you need to create a post process material for controlling the alpha channel. Create a new material in your project.

Open the material in the Material Editor.

In Details > Material, select Post Process in the Material Domain field.

In Details > Post Process Material, enable Output Alpha.

Save the material and go back to the Details panel of the Post Process Volume.

Click + in Rendering Features > Post Process Materials > Array, select Asset reference in the dropdown appearing below it and select the post process material you just created.

Your post process material is now active and affecting the rendering. Now you just have to edit the material to achieve the desired end result. Open the post process material in the Material Editor.

Alpha channel can be controlled in many ways. This example shows how to show the image from video pass-through cameras in the background using scene depth. The Opacity output value is set 0, if SceneDepth > 100000 and the control parameter RealWorld is 1.0.

You can now control the post process effect by modifying the RealWorld parameter in a blueprint. When enabling the mixed reality background, you should also disable the sky sphere.