Mixed reality settings

This section of the documentation explains how you should handle settings related to the video pass-through cameras.

On this page you could find following topics:

Occluding real world objects

As previously mentioned, by default, the virtual pass-through layer is always behind VR regardless of where in the physical world the objects are located. That means you won’t see your hands through an object located a few meters away even if you raise your hands close to your face. Varjo XR-3 utilizes color and infrared cameras to estimate the depth of real-world objects in real-time.

To enable depth estimation, call this:

varjo_MRSetVideoDepthEstimation(m_session, varjo_True); 

Then submit VR application depth and toggle depth testing by attaching a properly populated varjo_ViewExtensionDepth extension for the layer. Submit the layer with varjo_LayerFlag_DepthTesting flag enabled. After this is done, your hands will be occluded when they are closer than the virtual object.

Varjo is are continuously improving its depth estimation software for higher quality results. In the future, depth maps will become available for developers as streams.

Changing default camera settings

By default, Varjo software adjusts cameras for optimal brightness and color tone. However, you can override those values. Through the API, you can change exposure, ISO/gain, white balance, and flickering compensation.

First, lock the camera config:

varjo_Bool wasLocked = varjo_MRLockCameraConfig(m_session)

Only one application can lock the config at a time, so it is possible that your application can’t obtain the lock in case someone else is currently holding it. If the configuration can’t be locked, any attempts to change the camera properties will fail.

When you are holding the lock, you can e.g. change the operating mode to manual and change the manual setting.

varjo_MRSetCameraPropertyMode(m_session, varjo_CameraPropertyType_ExposureTime varjo_CameraPropertyMode_Manual);
varjo_MRSetCameraPropertyValue(m_session, varjo_CameraPropertyMode_Manual, propertyValue);

Finally, unlock the camera config. Alternatively, if you want to prevent other applications from changing the camera properties, you can leave the configuration locked.

varjo_MRUnlockCameraConfig(m_session)

You can determine the supported modes and values with:

varjo_MRGetCameraPropertyModeCount()
varjo_MRGetCameraPropertyModes()
varjo_MRGetCameraPropertyValueCount()
varjo_MRGetCameraPropertyValues()

Auto exposure presets

Default auto exposure is intended for general use and is more comfortable in wide range of sitautions.

PreserveHighlights preset offers a tuning, where auto exposure reacts quicker and more aggressively to any oversaturation in the image.

Eye reprojection

There is a physical offset between the user’s eyes and the video pass-through cameras, which can cause the perception of the real world to feel different than without the HMD. When the eye reprojection is turned on, the software reprojects the video pass-through image from the physical camera position into the physical position of the user’s eyes.

While this will improve the perception, it can also cause visual artifacts and therefore can be unwanted behavior for some applications.

This property offers two different reprojection modes:

When mode is Auto, the view is automatically reprojected based on the depth map. This is the most physically accurate reprojection.

When mode is Manual, the view is reprojected into a certain static distance. The value is a float specifying the reprojection distance in meters.

The eye reprojection property is changed the same way as other camera settings.

Getting RAW camera images

The API allows you to obtain raw images from the camera for your application. This is useful if your application wants to perform e.g. OpenCV operations on the images. The resolution of the image will be lower than what you are able to see through VST.

Note: Do not draw this image as it will incur latency.

See the DataStreamer file in the MRExample for the code example.