Chroma key

Chroma keying is a video technique where a predefined color is replaced with virtual content. Usually the color that is replaced is bright green or bright blue, as those colors contrast the most with human skin.

We have added following new functionality into the native API:

  • Enabling/disabling chroma keying for video pass-through image.
  • Managing configuration parameters for the chroma keying.
  • Controlling client application layer masking by adding new layer flag.

Chroma keying API can be used for multiple use cases such as simulators. One example is to have real world physical controls and cockpit and have a green/blue screen for the outside world VR projection. Another option is to have physical real-world controls (e.g. joystick, throttle, steering wheel, pedals) in green/blue screen environment while the cockpit and outside world is rendered in VR. There are other use cases as well, e.g. masking virtual touch displays in the cockpit. The Chroma key feature makes it possible for multiple users to experience the same virtual content in a shared green screen environment.

Note: We recommend enabling eye tracking when using chroma keying. With eye tracking enabled user would get more accurate results of chroma key calculations and mask filtering.

A simplified version of the chroma key API is implemented in Varjo Base. The chroma key UI in Varjo Base provides an easy way to experiment with the feature. For more complex applications, the chroma key API provides more functionality and flexibility.

Table of Contents

Features

  • Chroma keying feature toggle.
  • Chroma keying is active if at least one client has requested it. Chroma keying is global feature and will be done according to the active configurations.
  • Configurations:
    Chroma key configurations are indexed for supporting multiple configurations. In first phase all configurations are combined to a single mask.
  • Lock / unlock:
    Client must acquire a lock for modifying chroma keying parameters. A lock can be acquired for a short time to adjust parameters, or, for example, for the full lifetime of the application. Configurations can be read without a lock.
  • Get / Set: Functionality to read and write chroma key parameters.
  • Event varjo_EventType_MRChromaKeyConfigChange is sent whenever the chroma key configuration has changed outside the application control. A good case example is the user changing the chroma key configuration in Varjo Base.

MR API

The public API will have functionality for toggling the chroma key feature on or off, managing chroma key configurations, and operating automatic config estimator.

Functions:

// Toggle chroma key feature on/off 
VARJO_API void varjo_MRSetChromaKey(struct varjo_Session* session, varjo_Bool enabled); 

// Lock chroma keying parameters for application 
VARJO_API varjo_Bool varjo_MRLockChromaKeyConfig(struct varjo_Session* session); 

// Unlock chroma keying parameters 
VARJO_API void varjo_MRUnlockChromaKeyConfig(struct varjo_Session* session); 

// Return supported chroma key config count 
VARJO_API int32_t varjo_MRGetChromaKeyConfigCount(struct varjo_Session* session); 

// Set chroma key configuration for given index 
VARJO_API void varjo_MRSetChromaKeyConfig(struct varjo_Session* session, int32_t index, struct varjo_ChromaKeyConfig* config); 

// Returns chroma key configuration for given index 
VARJO_API struct varjo_ChromaKeyConfig varjo_MRGetChromaKeyConfig(struct varjo_Session* session, int32_t index);

Types:

// Chroma keying parameters for HSV colorspace chroma key matcher 
struct varjo_ChromaKeyParams_HSV; 

//  Wrapper for different chroma key matcher parameters 
union varjo_ChromaKeyParams; 

// Chroma key config structure including type info and actual parameters. 
struct varjo_ChromaKeyConfig;

Layer API

Chroma keying (if enabled by varjo_MRSetChromaKey) is only applied to client layers that have set the varjo_LayerFlag_ChromaKeyMasking flag.

The following flag needs to be added to layer flags:

// Layer is masked by VST chroma key if feature enabled 
static const varjo_LayerFlags varjo_LayerFlag_ChromaKeyMasking = 0x20;

Things to consider

  • The Chroma key feature has an impact on overall system performance, as video pass-through processing is using extra resources.
  • As a technique, chroma keying requires controlled environment and lighting. All shadows and reflections are bad, constant lighting is good.
  • Chroma keying is sensitive to flaws in video pass-through camera picture, especially chroma noise. Chroma keying works best with as low as possible camera ISO modes in well lit environment.