Creating realistic MR

Note: you are currently viewing documentation for a beta or an older version of Varjo

This page explains methods for how you can create realistic-looking virtual objects in mixed reality.

On this page you could find following topics:

Matching VR output with video signal

For a realistic MR experience, the brightness and white balance of VR output should match those of the video pass-through image.

Subscribe to the DistortedColor stream which provides constantly updated camera parameter data. See the code example in MRExample -> DataStreamer.cpp

To match the brightness, configure the camera exposure settings in your VR content renderer to match the video pass-through cameras. The luminance (in cd/m2) which saturates a pixel in the video pass-through image is equal to 2ev * cameraCalibrationConstant. See the code example in MRExample -> Scene.cpp

An alternative example for Unity HDRP can be found in the Unity XR plugin samples.

White balance

The DistortedColor stream metadata provides complete white balance normalization data to match VR colors and HDR cubemap with the video pass-through image.

VR content should be rendered with a fixed 6500K white point. White balance normalization can then be applied as a post-process step using the following formula:

finalColor = saturate(saturate(color * invCCM * diag(whiteBalanceColorGains)) * ccm)

where color is a 3D row vector, saturate(x) means clamping out-of-bounds values to the [0, 2B-1] range assuming B-bit output image, and diag(x) is a diagonal matrix with x values on the diagonal.

Illuminating virtual objects with real-world lighting

For realistic rendering, it is important to match the lighting of virtual objects with the real environment.

Subscribe to the EnvironmentCubemap stream which provides constantly updated environment lighting data. See the code example in MRExample -> DataStream.cpp

In the 2.4.0 release, the following changes were introduced:

  • The cubemap is now provided in HDR and contains accurate luminance for both direct and indirect sources of light.

  • The HDR cubemap now contains the environment’s luminance in cd/m2 with a 1:100 ratio. For example, a value of RGB=(1, 1, 1) corresponds to a luminance of 100 cd/m2.

  • The white point of the cubemap has been normalized to 6500K. See the previous chapter on how to match the white balance with the current video pass-through settings.

An example for Unity HDRP can be found in the Unity XR plugin samples.

Rendering shadows from virtual environments to real world

Since you might not have a virtual environment for your MR application, your virtual objects will not cast shadows.

To correct this, you can model a floor for your environment in the same position as the real floor. Typically it’s enough to render a plane at floor level. After that, you can render a black VR color and write the shadow intensity into the alpha channel.