Eye tracking

USING THE EYE TRACKING DATA

You can access per frame tracked eye tracking data through the following functions:

  • struct varjo_Gaze varjo_GetGaze(struct varjo_Session*)
    • returns current state of user’s gaze
  • varjo_Bool varjo_GetGazeData(struct varjo_Session*, struct varjo_Gaze*, struct varjo_EyeMeasurements*)
    • returns current state of user’s gaze and gaze tracker’s measurements of user’s eyes related properties
  • int32_t varjo_GetGazeArray(struct varjo_Session*, struct varjo_Gaze*, int32_t)
    • returns gaze tracking states for frames since previous query
  • int32_t varjo_GetGazeDataArray(struct varjo_Session*, struct varjo_Gaze*, struct varjo_EyeMeasurements*, int32_t)
    • returns gaze tracking states and gaze tracker’s measurements of user’s eyes related properties for frames since previous query

You can also access some eye tracking related state information and user IPD estimate using varjo property interface functions.

varjo_Gaze

  • varjo_Ray leftEye
  • varjo_Ray rightEye
  • varjo_Ray gaze
  • double focusDistance
  • double stability
  • varjo_Nanoseconds captureTime
  • varjo_GazeEyeStatus leftStatus
  • varjo_GazeEyeStatus rightStatus
  • varjo_GazeStatus status
  • int64_t frameNumber
  • [deprecated] double leftPupilSize
  • [deprecated] double rightPupilSize

varjo_Ray contains eye position coordinates [origin (x, y, z)] and gaze direction [forward (x, y, z)]. This data is recorded for the left eye (leftEye), the right eye (rightEye), and their weighted combination (gaze). It is given in the left-hand coordinate system (X points right, Y points top, Z points forward) and relative to the head pose. Please note that not only forward, but also origin component of the returned varjo_Ray must be accounted when computing the gaze ray projection to a 3D scene. We model the eyes so that the left eye ray origin is returned as (-halfIPD, 0, 0) and the right eye ray origin is returned as (+halfIPD, 0, 0), where halfIPD is half of the current interpupillary distance (IPD) of the headset in meters.

focusDistance returns the distance between the eye and the focus point. It has a value between 0.0 and 2.0 meters.

stability returns the stability of the user’s focus. It has a value between 0.0 and 1.0, where 0.0 indicates the least stable focus and 1.0 the most stable.

captureTime captureTime returns a timestamp, in nanoseconds, of when the eyes video frame was recorded. It is relative to the Varjo system time epoch.

leftStatus and rightStatus return a value for each eye as follows:

  • 0 – eye is not tracked and not visible (e.g., the eye is shut)
  • 1 – eye is visible but not reliably tracked (e.g., during a saccade or blink)
  • 2 – eye is tracked but quality is compromised (e.g., the headset has moved after calibration)
  • 3 – eye is tracked

status returns a value for the status of eye tracking in the Varjo headset as follows:

  • 0 – data unavailable; user is not wearing the device or eyes cannot be found
  • 1 – user is wearing the device, but gaze tracking is being calibrated
  • 2 – data is valid

frameNumber returns a unique identifier of the frame when the data was recorded.

leftPupilSize and rightPupilSize return values for the size of the pupil. The pupil size is measured in pixels from the eye tracking camera and then normalized to 0.0…1.0 range via a division to the maximum tracked pupil size constant. These measurements are deprecated and will be removed in a future release. New code should use data from varjoEyeMeasurements instead.

varjo_EyeMeasurements

  • int64_t frameNumber
  • varjo_Nanoseconds captureTime
  • float interPupillaryDistanceInMM
  • float leftPupilIrisDiameterRatio
  • float rightPupilIrisDiameterRatio
  • float leftPupilDiameterInMM
  • float rightPupilDiameterInMM
  • float leftIrisDiameterInMM
  • float rightIrisDiameterInMM
  • float leftEyeOpenness
  • float rightEyeOpenness

interPupillaryDistanceInMM returns an estimate of the user’s inter-pupillary distance measured in millimeters. 0.0 if an estimate is not available.

leftPupilIrisDiameterRatio and rightPupilIrisDiameterRatio return ratio of the user’s pupil diameter estimate to an estimated iris diameter. 0.0 if either estimate is not available and the ratio cannot be calculated.

leftPupilDiameterInMM and rightPupilDiameterInMM return an estimated diameter of the tracked pupils in millimeters. If the pupil is not tracked or the value is otherwise not available, 0.0 is returned.

leftIrisDiameterInMM and rightIrisDiameterInMM return an estimated diameter of the tracked irises in millimeters. These estimations are updated when the user puts the headset on the head. The estimated values are valid until the user removes the headset from the head. 0.0 is returned when the estimate is not available.

leftEyeOpenness and rightEyeOpenness return estimated openness ratios of the eyes. Value 1.0 corresponds to a fully open eye, 0.0 to a fully closed eye.

Please note that leftPupilIrisDiameterRatio, rightPupilIrisDiameterRatio, leftPupilDiameterInMM, rightPupilDiameterInMM, leftIrisDiameterInMM, rightIrisDiameterInMM, leftEyeOpenness, rightEyeOpenness are experimental features and provided in this release without any accuracy guarantees.

Gaze Tracking Output Options

When using the Varjo gaze tracking system via the public API, you can choose between a smoothed (default) and raw gaze tracking output and from maximum supported (default), 100Hz and 200Hz tracking data rate options.

For the default gaze tracking settings, call the varjo_GazeInit function with no arguments.

Note: behavior of varjo_GazeInit changed starting from the Varjo Base 4.1 release. Previously the gaze tracking output frequency was initialized to 100Hz by default, and starting from 4.1 it is initialized to the maximum supported gaze tracking output frequency of the connected headset. Some applications that expect the reduced tracking rate might require code fix due to this change. In a such case application should explicitly set output frequency parameter to 100Hz, or be modified other ways to handle the maximum gaze tracking data rate.

For custom gaze tracking output settings, use the varjo_GazeInitWithParameters function with the varjo_GazeParameters argument filled according to your desired output mode:

  • Set OutputFilterType to Standard for smoothed output and None for raw output.
  • Set OutputFrequency to OutputFrequency100Hz, OutputFrequency200Hz, or OutputFrequencyMaximumSupported (for the maximum data rate supported by connected headset).

It is possible to call varjo_GazeInit and varjo_GazeInitWithParameters both before and after the headset is connected. Initializing the gaze with an unsupported data stream frequency can lead to error varjo_Error_UnsupportedParameter, if the headset is connected at the time of the initialization. If the headset is not yet connected, the data stream frequency support is checked again when the headset is connected, which might result in varjo_Error_UnsupportedParameter error on the next call to any API function retrieving new gaze data.

Data Stream

Native Varjo DataStream API can be used to access eye camera video stream in real time.

Following constant specifies the stream type that is reserved for eye camera data stream

/**
 * Distorted (i.e. uncorrected) luminance data stream from eye tracking monochrome camera.
 *
 * Frames contain the following additional data for each channel:
 * - Buffers (access with #varjo_GetBufferId)
 */
static const varjo_StreamType varjo_StreamType_EyeCamera = 3;

When a frame from the eye camera data stream is received, its varjo_StreamFrameMetadata will contain metadata defined by following structure.

/**
 * Metadata for eye camera stream.
 *
 * This metadata is available in #varjo_StreamFrameMetadata structure
 * when #varjo_StreamFrame::type equals #varjo_StreamType_EyeCamera.
 *
 * Glint LEDs are numbered in clockwise direction for left eye and counter clockwise for right
 * eye starting from middle nose side glint LED. Refer to developer documentation for more
 * information.
 */
struct varjo_EyeCameraFrameMetadata {
    varjo_Nanoseconds timestamp;  //!< Timestamp when the frame was captured (start of frame).
    uint32_t glintMaskLeft;       //!< Bit mask of enabled glint LEDs of left chamber
    uint32_t glintMaskRight;      //!< Bit mask of enabled glint LEDs of right chamber
};

See “DataStream API documentation” and ”EyeCameraStreamExample” for more information.

Calibration

There are two types of calibration available in the Varjo eye tracking system: Fast and OneDot calibrations.

  • Fast calibration is a quick five-dot calibration that uses observations from the current user as well as statistical priors generated from diverse, previously collected and processed observations from other people.
  • OneDot calibration is the quick calibration that is also used when the headset is automatically calibrated for foveated rendering. Like Fast calibration, OneDot uses observations from the current user and statistical priors to construct calibration with adequate accuracy.

After a successful calibration, the eye tracking data is immediately available via the API functions retrieving gaze data

Note that support for Legacy gaze calibration was removed in Varjo Base 3.10. Newer Fast gaze calibration allows to achieve higher accuracy and precision. Requesting Legacy gaze calibration by application will trigger Fast calibration starting from Varjo Base 3.10.

Automatically Triggered Calibration

The automatic OneDot calibration is supported for headsets that use foveated rendering: XR-4, XR-3, VR-3 and Aero. Foveation Calibration Mode setting in Varjo Base UI controls when this automatic gaze calibration is done:

  • in Always calibrate mode OneDot calibration is launched automatically every time the user puts on the headset
  • in Remember my calibration mode OneDot calibration is launched only when the user puts on the headset for the first time; the data is saved and reused in all the following sessions
  • OneDot calibration is not launched automatically in Best estimation without calibration and Never calibrate modes

Manually Triggered Calibration

OneDot gaze calibration can be triggered manually from the Varjo Base UI via Calibrate eye tracking button. Fast calibration, which takes longer time to complete but results in a more accurate gaze estimation, can be launched from Analytics window.

Requesting Calibration Programmatically

The gaze calibration can be triggered programmatically with the varjo_RequestGazeCalibration and varjo_RequestGazeCalibrationWithParameters functions. Choose between Fast (default) and OneDot calibration by filling the GazeCalibrationType parameter appropriately. Calling varjo_RequestGazeCalibration will trigger Fast calibration.

/**
 * Requests an HMD gaze calibration.
 *
 * This attempts to trigger the gaze calibration sequence if the user has
 * allowed gaze tracking from Varjo settings and Varjo system is in a state
 * where it can bring up the calibration UI.
 *
 * @param session Varjo session handle.
 */
VARJO_API void varjo_RequestGazeCalibration(struct varjo_Session* session);

/**
 * Requests an HMD gaze calibration with provided parameters.
 *
 * This attempts to trigger the gaze calibration sequence if the user has
 * allowed gaze tracking from Varjo settings and Varjo system is in a state
 * where it can bring up the calibration UI.
 *
 * @param session Varjo session handle.
 * @param parameters Gaze calibration key-value parameters.
 * @param parameterCount Number of parameters provided.
 */
VARJO_API void varjo_RequestGazeCalibrationWithParameters(
    struct varjo_Session* session, struct varjo_GazeCalibrationParameters* parameters, int32_t parameterCount);

Note that requesting the gaze calibration when the headset is not connected will result in error varjo_Error_GazeNotConnected.

Calibration Parameters

varjo_RequestGazeCalibrationWithParameters function accepts two parameters that control the gaze calibration procedure:

  • GazeCalibrationType parameter is used to control what type of calibration should be done. Possible values are Fast (default) and OneDot. See their descriptions here

  • HeadsetAlignmentGuidanceMode parameter controls how the headset alignment guidance UI should behave. Possible values are:

    • WaitInput (default) that causes the calibration UI to wait for the user to confirm the acceptable headset alignment position via pressing the headset button or “Calibrate eye tracking” menu button in Varjo Base before continuing to gaze calibration.
    • Automatic that causes UI to continue automatically to the actual calibration after a headset alignment has been detected as acceptable by Varjo Base calibration UI.

Cancelling Calibration Programmatically

The gaze calibration procedure can be cancelled anytime calling varjo_CancelGazeCalibration function. This will close the gaze calibration UI and return the gaze tracker to its initial state. Note that the function can be called also when the calibration UI is not visible to reset the active calibration to defaults.

When the gaze calibration is cancelled in Always calibrate, Remember my calibration and Best estimation without calibration modes, Best estimation without calibration will still be used to estimate the user’s gaze. When the calibration is cancelled in Never calibrate mode, the user’s gaze direction is not estimated.

/**
 * Cancels the currently active gaze calibration routine, if any, and resets the gaze tracker to a default state
 *
 * After this function has been called, any active gaze calibration user
 * interface will be closed. The last successful gaze calibration, if any, will
 * be reset. The gaze tracker will continue without calibration and may still
 * estimate the gaze using "Best estimation without calibration" depending on
 * the foveated rendering calibration mode currently selected in Varjo Base.
 *
 * @param session Varjo session handle.
 */
VARJO_API void varjo_CancelGazeCalibration(struct varjo_Session* session);

Property interface for eye tracking

Following eye tracking related properties can be accessed using varjo_GetPropertyBool and varjo_GetPropertyDouble functions.

  • bool varjo_PropertyKey_UserPresence - Is user wearing the HMD.
  • bool varjo_PropertyKey_GazeCalibrating - Is system currently calibrating the gaze tracker.
  • bool varjo_PropertyKey_GazeCalibrated - Is the gaze tracker calibrated.
  • bool varjo_PropertyKey_GazeAllowed - Is the application allowed to use the HMD gaze tracker data.
  • double varjo_PropertyKey_GazeCalibrationQuality_Left - Quality assessment of the left eye gaze calibration. In range [0.0-1.0].
  • double varjo_PropertyKey_GazeCalibrationQuality_Right - Quality assessment of the right eye gaze calibration. In range [0.0-1.0].
  • double varjo_PropertyKey_GazeIPDEstimate - Estimate of the user’s interpupillary distance in millimeters.

An example on how to use the property interface:

// Sync properties from runtime. Call this e.g. once per frame.
varjo_SyncProperties(session);

// Now access properties
bool gazeAllowed = varjo_GetPropertyBool(session, varjo_PropertyKey_GazeAllowed);
double userIpd = varjo_GetPropertyDouble(session, varjo_PropertyKey_GazeIPDEstimate);

Privacy Settings

Varjo Base UI has a privacy setting toggle “Allow eye tracking“ that impacts the behavior of the Varjo eye tracking native API.

In case “Allow eye tracking” is off, calling varjo_RequestGazeCalibration, varjo_RequestGazeCalibrationWithParameters or any function retrieving new gaze data will result in error varjo_Error_GazeNotAllowed.

It is possible to query the state of this Base setting via varjo_IsGazeAllowed function.

Please note that this privacy setting does not prevent Varjo system to launch quick one-dot Foveation calibration and use eye tracking for foveated rendering.

Examples

Please check ”GazeTrackingExample” application that demonstrates usage of Varjo native API for gaze tracking, and ”EyeCameraStreamExample” application that demonstrates how to access eye camera stream using native Varjo data stream API.