Creating realistic MR
This page explains methods for how you can create realistic-looking virtual objects in mixed reality.
On this page you could find following topics:
- Matching VR output with video signal
- White balance
- Illuminating virtual object with real world lighting
- Rendering shadows from virtual environments to real world
Matching VR output with video signal
For a realistic MR experience, the brightness and white balance of VR output should match those of the video pass-through image.
Subscribe to the DistortedColor stream which provides constantly updated camera parameter data. See the code example in MRExample -> DataStreamer.cpp
To match the brightness, configure the camera exposure settings in your VR content renderer to match the video pass-through cameras. The luminance (in cd/m2) which saturates a pixel in the video pass-through image is equal to
2ev * cameraCalibrationConstant. See the code example in MRExample -> Scene.cpp
An alternative example for Unity HDRP can be found in the Unity XR plugin samples.
The DistortedColor stream metadata provides complete white balance normalization data to match VR colors and HDR cubemap with the video pass-through image.
VR content should be rendered with a fixed 6500K white point. White balance normalization can then be applied as a post-process step using the following formula:
finalColor = saturate(saturate(color * invCCM * diag(whiteBalanceColorGains)) * ccm)
where color is a 3D row vector, saturate(x) means clamping out-of-bounds values to the [0, 2B-1] range assuming B-bit output image, and diag(x) is a diagonal matrix with x values on the diagonal.
Illuminating virtual objects with real-world lighting
For realistic rendering, it is important to match the lighting of virtual objects with the real environment.
Subscribe to the EnvironmentCubemap stream which provides constantly updated environment lighting data. See the code example in MRExample -> DataStream.cpp
In the 2.4.0 release, the following changes were introduced:
The cubemap is now provided in HDR and contains accurate luminance for both direct and indirect sources of light.
The HDR cubemap now contains the environment’s luminance in cd/m2 with a 1:100 ratio. For example, a value of RGB=(1, 1, 1) corresponds to a luminance of 100 cd/m2.
The white point of the cubemap has been normalized to 6500K. See the previous chapter on how to match the white balance with the current video pass-through settings.
An example for Unity HDRP can be found in the Unity XR plugin samples.
Rendering shadows from virtual environments to real world
Since you might not have a virtual environment for your MR application, your virtual objects will not cast shadows.
To correct this, you can model a floor for your environment in the same position as the real floor. Typically it’s enough to render a plane at floor level. After that, you can render a black VR color and write the shadow intensity into the alpha channel.