Masking

When developing for Varjo XR headsets, you can mask specific objects to act as a “window” into the real world. This is especially useful if you want to see real-world elements, such as instruments or controller devices, in the virtual environment.

Masking a real world object

This example shows a scene where the user can sit inside a Volvo XC40 Recharge car. They are able to simultaneously see the physical car (including the steering wheel, dashboard, and doors) and virtual content outside of the car (e.g., roads, a garage, or a studio).

Start by creating a 3D model of the parts of the scene that you would like see in the video pass-through image. This model will act as the mask.

  • In this case we have removed the windows, so that they will not block the virtual content from being visible on the outside of the car.

  • Only the silhouette of the mask is important. Details that do not affect the silhouette from the user’s viewpoint can be ignored or removed from the 3D model.

Import the 3D model into the project and apply the video pass-through material to it. See Masking for more information.

The 3D model must be precisely aligned with the corresponding real-world element, including its position, rotation, and scale.

This can be accomplished in three ways:

  • Using Varjo Markers
  • Using External trackers such as HTC Vive trackers
  • Manual alignment through code.

Adding virtual content to the real world

This example shows a scene where the user is able to see the world around them via video pass-through with a virtual object added in.

First, enable video pass-through in your project for the background.

Import your 3D assets and set up their desired materials and look. Ensure that both the XR camera rig and the 3D assets are at the correct height relative to the ground.

Adding real-time environmental reflections allows reflections in the video pass-through image to affect the virtual object, making it blend in better with the real world. You can enable this in the Varjo XR plugin. See Environment reflections in Unity for an example in Unity.

Adding shadows to the ground under your virtual objects can make them look more grounded and convincing. For static objects, you can bake the shadows onto a texture in an external program and then place it onto a ground plane. The texture should only have the shadow information with other parts being transparent. For dynamic scenes, add a Shadow Catcher to a plane at ground level. See Shadow catcher in Unity for an example in Unity.