Mixed Reality with Unreal Engine 5

Developing mixed reality applications for Varjo XR headsets is quick and easy with Varjo OpenXR plugin. Follow these instructions to get started.

Once you have set up the Varjo OpenXR plugin (see Getting Started with Unreal) follow these instructions to enable mixed reality in your project.

Create a new blueprint class from the Actor parent class called BP_MRControls.

Drag the blueprint into the Scene.

In BP_MRControls, create variable VRPawn > set Variable Type to VRPawn and enable Private.

Recreate the logic from the reference image.

Go to Project Settings > Engine - Input > Action Mappings > create MRToggle and set it to key M.

Please note that later versions of Unreal 5 have deprecated Action Mappings and it’s recommended you create an Enhanced Input Action instead.

In BP_MRControls in Event Graph > right-click and select MRToggle.

Use IsMixedRealitySupported blueprint node to determine if mixed reality is available for the currently connected device.

Use IsMixedRealityEnabled blueprint node to check if mixed reality is currently enabled.

Create a Execute Console Command node, set it to run the command xr.OpenXREnvironmentBlendMode 3 (= Alpha Blend) to enable video pass-through mixed reality.

Create a Execute Console Command node, set it to run the command xr.OpenXREnvironmentBlendMode 1 (= Opaque) to disable it. Mixed reality is enabled by default for mixed reality capable Varjo devices. Starting from Unreal Engine 5.0.2, you can enable and disable mixed reality in runtime. Mixed reality can be controlled using the xr.OpenXREnvironmentBlendMode console variable.

Note: Unreal Engine 5 doesn’t currently do any validation for xr.OpenXREnvironmentBlendMode. Use Varjo-provided functions to determine if mixed reality is supported with the connected device.

A simple blueprint for controlling mixed reality could look like this:

Recreate the logic from the reference image

The easiest way to control the alpha channel is by using custom postprocessing. First you need to enable the alpha channel. In Project Settings > Rendering > Postprocessing, set Enable alpha channel support in post processing to Allow through tonemapper.

Note: If your project is using quality setting level Epic or higher, the used color format (PF_FloatRGBA 64Bit) has an additional alpha channel and it may cause you to see only video pass-through image in the headset. To fix this, either change Effects to level High in Engine Scalability Settings or set r.SceneColorFormat 3 (PF_FloatRGB 32Bit).

To set r.SceneColorFormat > select Open Level Blueprint > connect Event BeginPlay to Execute Console Command and add r.SceneColorFormat 3. Compile and Save.

Additionally, you can add an Exit logic.

Add a Post Process Volume in your scene and enable Infinite Extent (Unbound) under Post Process Volume Settings in the Details panel.

Next, you need to create the post process material for controlling the alpha channel. Create a new Material in your project and name it PP_MR.

Open the material in the Material Editor.

In Details > Material, select Post Process in the Material Domain field.

In Details > Post Process Material, enable Output Alpha.

Save the material and go back to the Details panel of the Post Process Volume.

Click + in Rendering Features > Post Process Materials > Array, select Asset reference in the dropdown appearing below it and select the post process material you just created.

Your post process material is now active and affecting the rendering. Now you just have to edit the material to achieve the desired end result. Open the post process material in the Material Editor.

Alpha channel can be controlled in many ways. This example shows how to show the image from video pass-through cameras in the background using scene depth. Recreate the logic from the reference image

If you disable sky sphere, you should now be able to see the mixed reality in the background. In this example we have a VR Only Objects list, which contains the sky sphere and other objects we want to hide in mixed reality mode. When toggling the mixed reality background, we simply iterate through this list and set the visibility.

To create a Background Toggle logic, go to Project Settings > Engine – Input > Action Mappings > create MRBackgroundToggle and set it to key B (or use Enhanced Input Actions instead for later Unreal 5 versions).

Go back to BP_MRControls and create MRBackgroundToggle.

Create a variable MRBackgroundEnabled and set Variable Type to Boolean. Create a variable VROnlyObjects and make it Public by setting it to Actor – Array.

To view the result, go to Viewport in the Outliner and select BP_MRControls. In BP_MRControls, go to Details > Defaults >VROnly Objects and click + to select the objects that won’t be visible in XR (SkySphere, walls, etc).