Understading advanced multi-app cases

Varjo compositor supports advanced ways of using mixed reality and blending multiple applications or rendering layers together. These cases can become very complex and hard to understand. This page illustrates some cases and combinations that Varjo supports.

Table of Contents

• Application render order
  • Priority
  • OpenXR / OpenVR Priority
  • Shared priority
  • Layers
• Mixed reality
  • Mixed reality (alpha)
  • Mixed reality (alpha) with multiple layers / applications
  • Mixed reality (blend control mask)
  • Mixed reality (blend control mask) + Mixed reality (alpha)
• Chroma key
  • Chroma key (in application)
  • Chroma key (global)
  • Chroma key (blend control mask)
  • Chroma key (blend control mask) + Mixed reality (alpha)
• Depth occlusion
  • Depth occlusion (in application)
  • Depth occlusion (global)
  • Depth occlusion (blend control mask)
  • Depth occlusion (global) + Chroma key (blend control mask)
• Rendering application + Utility application
• Notes

Application render order

Priority

Applications are rendered in priority order. Priority can be set with Varjo native SDK. Higher priority applications are rendered on top. You can render multiple Varjo native applications at the same time. Transparent pixels in the top application, show the application underneath.

Cockpit application	Terrain application	Final image
Priority 0 Varjo native / OpenXR / OpenVR	Priority 1 Varjo native	The Terrain application is rendered on top of the Cockpit application.

Terrain application	Cockpit application	Final image
Priority 0 Varjo native / OpenXR / OpenVR	Priority 1 Varjo native	The Cockpit application is rendered on top of the Terrain application.

OpenXR / OpenVR Priority

Applications are rendered in priority order. Priority cannot be set with OpenXR or OpenVR SDK. OpenXR and OpenVR applications use always priority 0. You can only run one OpenXR and OpenVR application at a time.

Terrain application	Cockpit application	Final image
Priority 0 (cannot be changed) OpenXR / OpenVR	Priority 1 Varjo native	The Cockpit application is rendered on top of the Terrain application.

Terrain application	Cockpit application	Final image
	ERROR
Priority 0 (cannot be changed) OpenXR / OpenVR	Priority 0 (cannot be changed) OpenXR / OpenVR	The Terrain application is only rendered. The Cockpit application cannot render anything.

Shared priority

Applications sharing the same priority are not rendered on top of each other. Only one of the applications is rendered.

Terrain application	Cockpit application	Final image
Priority 0 Varjo native / OpenXR / OpenVR	Priority 0 Varjo native / OpenXR / OpenVR	Only the Terrain application is rendered.

Layers

A single application can submit multiple layers for rendering. Layers can be manually ordered within the application. All layers will share the priority of the application submitting them.

Flightsim application Terrain layer	Flightsim application Cockpit layer	Flightsim application HUD layer	Final image
Priority 0 Layer 1 Varjo native	Priority 0 Layer 2 Varjo native	Priority 0 Layer 3 Varjo native	The Cockpit layer is rendered on top of the Terrain layer within the Flightsim application.

Mixed reality

Mixed reality (alpha)

Mixed reality is always rendered behind everything. Transparent pixels in the top application, show the real world. This is called an alpha mask.

Varjo compositor	Flightsim application	Final image
Video see-through	Priority 0 Alpha mask Varjo native / OpenXR	The Flighstim application is rendered on top of the video see-through.

Mixed reality (alpha) with multiple layers / applications

Multiple layers have to submit transparent pixels, for real world to show through. Otherwise only the application underneath is shown.

Varjo compositor	Terrain application	Cockpit application	Final image
Video see-through	Priority 0 Varjo native / OpenXR / OpenVR	Priority 1 Alpha mask Varjo native	Only the Terrain application is rendered below the Cockpit application.

Varjo compositor	Terrain application	Cockpit application	Final image
Video see-through	Priority 0 Alpha mask Varjo native / OpenXR	Priority 1 Alpha mask Varjo native	Video see-through is rendered to regions where both layers above have mask in the same location. Terrain is rendered to regions that only the Cockpit application is masking.

Varjo compositor	Terrain application	Cockpit application	HUD application	Final image
Video see-through	Priority 0 Alpha mask Varjo native / OpenXR	Priority 1 Alpha mask Varjo native	Priority 2 Alpha mask Varjo native	Video see-through is rendered to regions where all layers above have mask in the same location. Terrain is rendered to regions that only the Cockpit application is masking while the HUD application is rendered on top of everything.

Mixed reality (blend control mask)

Blend control mask is a special layer that injects mixed reality to all other layers and applications. It is always rendered on top, punching through to the real world view. Priority is ignored. Blend control masks are exclusive to Varjo native SDK. Varjo Lab Tools uses blend control masks.

Blend control mask is a global feature and only one layer can submit it. If more blend control masks layers are detected, they are ignored.

Varjo compositor	Flightsim application	Varjo Lab Tools	Final image
Video see-through	Priority 0 Varjo native / OpenXR / OpenVR	Priority ignored Blend control mask Varjo native	The Flightsim application is blended with video see-through image according to the blend control mask submitted by Varjo Lab Tools.

Varjo compositor	Flightsim application	HUD application	Varjo Lab Tools	Final image
Video see-through	Priority 0 Varjo native / OpenXR / OpenVR	Priority 1 Alpha mask Varjo native	Priority ignored Blend control mask Varjo native	The Flightsim application and HUD are blended with video see-through image according to the blend control mask submitted by Varjo Lab Tools. HUD gets cut off by the blend control mask.

Mixed reality (blend control mask) + Mixed reality (alpha)

Blend control masks can be used to add additional punch through holes that are not present in the alpha masking application.

Varjo compositor	Flightsim application	Varjo Lab Tools	Final image
Video see-through	Priority 0 Alpha mask Varjo native / OpenXR	Priority ignored Blend control mask Varjo native	The Flightsim application is blended with video see-through image according to the blend control mask submitted by Varjo Lab Tools and alpha masking included in the Flightsim application.

Chroma key

Chroma key (in application)

Applications using chroma key will be transparent from areas where-ever chroma key effect is applied. Chroma key is exclusive to Varjo native SDK.

Varjo compositor	Flightsim application	Final image
Video see-through	Priority 0 Varjo native Chroma key	Chroma key effect renders video pass-through in regions where it doesn't detect green.

Varjo compositor	Flightsim application	HUD application	Final image
Video see-through	Priority 0 Varjo native Chroma key	Priority 1 Varjo native Alpha mask	Chroma key effect renders video pass-through in regions where it doesn't detect green within the Flightsim application. The HUD is rendered on top.

Chroma key (global)

Global chroma key can be used to force chroma key on all layers and applications. Global chroma key is exclusive to Varjo native SDK, but Varjo Base provides global chroma key tool that can be easily used with all applications.

Varjo compositor	Flightsim application	Varjo Base	Final image
Video see-through	Priority 0 Varjo native / OpenXR / OpenVR	Chroma key Varjo native	Global chroma key effect renders video pass-through in regions where it doesn't detect green.

Varjo compositor	Flightsim application	HUD application	Varjo Base	Final image
Video see-through	Priority 0 Varjo native / OpenXR / OpenVR	Priority 1 Varjo native Alpha mask	Chroma key Varjo native	Chroma key effect renders video pass-through in regions where it doesn't detect green. Note that this will clip the HUD application as well.

Chroma key (blend control mask)

Chroma key can be used together with blend control masks. You can fine-tune where and how the chroma key effect is applied to.

Varjo compositor	Flightsim application	Varjo Lab Tools	Final image
Video see-through	Priority 0 Varjo native / OpenXR / OpenVR	Priority ignored Chroma key Blend control mask	Global chroma key effect renders video pass-through within the blend control mask where it sees green. Elsewhere render full virtual content.

Chroma key (blend control mask) + Mixed reality (alpha)

Blend control masks can be used to add additional punch through holes that are added to virtual content in chroma key room.

Varjo compositor	Flightsim application	Varjo Lab Tools	Final image
Video see-through	Priority 0 Alpha mask Varjo native / OpenXR	Priority ignored Chroma key Blend control mask	Global chroma key effect renders video pass-through within the blend control mask where it sees green. Application renders video pass-through to masked region. Elsewhere, render virtual content. Note that green instruments in the cockpit do not get chroma key effect in this scenario.

Depth occlusion

Depth occlusion (in application)

Depth occlusion creates transparency based on lidar. Whatever lidar sees, is transparent.

Varjo compositor	Flightsim application	Final image
Video see-through	Priority 0 Varjo native / OpenXR Depth occlusion	Hand visible to lidar is rendered visible on top of the Flightsim application.

Depth occlusion (global)

Global depth occlusion can be used to force depth occlusion on all layers and applications. Global depth occlusion is exclusive to Varjo native SDK, but it can be easily enabled with Varjo Lab Tools with all applications.

Varjo compositor	Flightsim application	Varjo Lab Tools	Final image
Video see-through	Priority 0 Varjo native / OpenXR / OpenVR	Varjo native Depth occlusion	Hand visible to lidar is rendered visible on top of the Flightsim application. Depth occlusion is forced on with Varjo Lab Tools.

Depth occlusion (blend control mask)

Depth occlusion can be used together with blend control masks. You can fine-tune where depth occlusion effects is applied to.

Depth occlusion (global) + Chroma key (blend control mask)

Chroma key and depth occlusion both can be used together with blend control masks. The behaviour of this combination is currently unspesified and may change in the future.

This is usually used when hands are wanted to be seen on top of virtual content in chroma key rooms.

Rendering application + Utility application

Not all Varjo features require rendering directly into the headset. In multi-app cases you can combine rendering applications with background utilities that control spesific features or just use read only data from the headset.

Utility examples:

• Headset mirror view
• Eye tracking analytics
• Eye camera view
• Video post process shader
• Mixed reality video feed

Varjo native SDK provides examples on how to access these. Varjo does not provide pre-built utilities outside of what Varjo Base and Lab Tools offers.

Varjo compositor	Flightsim application	Video post process utility	Final image
Video see-through	Priority 0 Alpha mask Varjo native / OpenXR	MR night vision	The Flighstim application is rendered on top of the video see-through. Video post process shader is only applied to video pass-through.

Varjo compositor	Flightsim application	Video post process utility	Final image
Video see-through	Priority 0 Alpha mask VR night vision Varjo native / OpenXR	MR night vision	The Flighstim application is rendered on top of the video see-through. Two night vision post process shaders are applied. One for virtual content and an another one for video pass-through.

Notes

When running multiple applications performance of the PC will be limited. By default Windows will prioritize the application that is selected currently. Utility applications generally, do not slow PC down as they don’t render anything into the headset.

Blend control mask punches through all the layers and application always. When blend control mask is used as any part of the rendering, you cannot render anything on top of them. No matter the priority. Ie. you cannot render a hud on top of mask that has been created with blend control mask. Alpha masks are ok.

If multiple applications try to use global features at a same time, only the most recent changes are applied unless the feature is locked and then only the application locking the feature can change them.

OpenVR does not have concept of mixed reality and is always fully opaque unless a global feature forces it to be in mixed reality.