Eye tracking

Note: you are currently viewing documentation for a beta or an older version of Varjo

USING THE EYE TRACKING DATA

You can acccess per frame tracked eye tracking data through following functions:

  • struct varjo_Gaze varjo_GetGaze(struct varjo_Session*)
    • returns current state of user’s gaze
  • varjo_Bool varjo_GetGazeData(struct varjo_Session*, struct varjo_Gaze*, struct varjo_EyeMeasurements*)
    • returns current state of user’s gaze and gaze tracker’s measurements of user’s eyes related properties
  • int32_t varjo_GetGazeArray(struct varjo_Session*, struct varjo_Gaze*, int32_t)
    • returns gaze tracking states for frames since previous query
  • int32_t varjo_GetGazeDataArray(struct varjo_Session*, struct varjo_Gaze*, struct varjo_EyeMeasurements*, int32_t)
    • returns gaze tracking states and gaze tracker’s measurements of user’s eyes related properties for frames since previous query

You can also access some eye tracking related state information and user IPD estimate using varjo property interface functions.

varjo_Gaze

  • varjo_Ray leftEye
  • varjo_Ray rightEye
  • varjo_Ray gaze
  • double focusDistance
  • double stability
  • varjo_Nanoseconds captureTime
  • varjo_GazeEyeStatus leftStatus
  • varjo_GazeEyeStatus rightStatus
  • varjo_GazeStatus status
  • int64_t frameNumber
  • [deprecated] double leftPupilSize
  • [deprecated] double rightPupilSize

varjo_Ray returns data about eye position coordinates [origin (x, y, z)] and vector direction [forward (x, y, z)]. Data is recorded for the left eye (leftEye), right eye (rightEye), and their weighted combination (gaze). Gaze data is given in the left-hand coordinate system (X points right, Y points top, Z points forward) and relative to head pose. Please note that not just forward, but also origin component of returned varjo_ray must be accounted when computing gaze ray projection to 3D scene.

focusDistance returns the distance between eye and focus point. It has a value between 0.0 and 2.0 meters.

stability returns the stability of the user’s focus. It has a value between 0.0 and 1.0, where 0.0 indicates least stable focus and 1.0 most stable.

captureTime captureTime returns a timestamp, in nanoseconds, of when the eyes video frame was recorded.

leftStatus and rightStatus return a value for each eye as follows:

  • 0 – eye is not tracked and not visible (e.g., the eye is shut)
  • 1 – eye is visible but not reliably tracked (e.g., during a saccade or blink)
  • 2 – eye is tracked but quality is compromised (e.g., the headset has moved after calibration)
  • 3 – eye is tracked

status returns a value for the status of eye tracking in the Varjo headset as follows:

  • 0 – data unavailable; user is not wearing the device or eyes cannot be found
  • 1 – user is wearing the device, but gaze tracking is being calibrated
  • 2 – data is valid

frameNumber returns a unique identifier of the frame when the data was recorded.

leftPupilSize and rightPupilSize return values for the size of the pupil. Pupil size is measured in pixels from eye tracking camera and then normalized to 0.0…1.0 range via division to the maximum tracked pupil size constant. These measurements are deprecated and will be removed in a future release. New code should use data from varjoEyeMeasurements instead.

varjo_EyeMeasurements

  • int64_t frameNumber
  • varjo_Nanoseconds captureTime
  • float interPupillaryDistanceInMM
  • float leftPupilIrisDiameterRatio
  • float rightPupilIrisDiameterRatio
  • float leftPupilDiameterInMM
  • float rightPupilDiameterInMM
  • float leftIrisDiameterInMM
  • float rightIrisDiameterInMM

interPupillaryDistanceInMM return estimate of user’s inter-pupillary distance measured in millimeters. 0.0 if estimate is not available.

leftPupilIrisDiameterRatio and rightPupilIrisDiameterRatio return ratio of user’s pupil diameter estimate to estimated iris diameter. 0.0 if either estimate is not available and ratio cannot be calculated.

leftPupilDiameterInMM and rightPupilDiameterInMM return estimated diameter of tracked pupils in millimeters. If pupil is not tracked or value is otherwise not available, 0.0 is returned.

leftIrisDiameterInMM and rightIrisDiameterInMM return estimated diameter of tracked irises in millimeters. These estimations are updated when user puts the headset on head. Estimate values are valid until user removes headset from head. 0.0 is returned when estimate is not available.

Please note, leftPupilIrisDiameterRatio, rightPupilIrisDiameterRatio, leftPupilDiameterInMM, rightPupilDiameterInMM, leftIrisDiameterInMM, rightIrisDiameterInMM are new experimental features and provided in this release without accuracy guaranties.

Gaze Tracking Output Options

When using the Varjo gaze tracking system via public API, you can choose between smoothed (default) and raw gaze tracking output and between 100Hz (default) and 200Hz tracking data rates.

For the default gaze tracking settings, call the varjo_GazeInit function with no arguments.

For custom gaze tracking output settings, use the varjo_GazeInitWithParameters function with the varjo_GazeParameters argument filled according to your desired output mode:

  • Set OutputFilterType to Standard for smoothed output and None for raw output.
  • Set OutputFrequency to OutputFrequency100Hz, OutputFrequency200Hz, or OutputFrequencyMaximumSupported (for the maximum data rate supported by connected headset).

It is possible to call varjo_GazeInit and varjo_GazeInitWithParameters both before and after headset is connected. Please note that OutputFrequency200Hz is not supported on older devices (VR-1, VR-2, VR-2 Pro, XR-1). Initializing gaze with unsupported data stream frequency can lead to error varjo_Error_GazeUnsupportedParameter, if headset is connected at the time of initialization. In the case when headset is not yet connected, data stream frequency support is checked again when headset is connected, which might result in varjo_Error_GazeUnsupportedParameter error on next call to any API function retrieving new gaze data.

Data Stream

Native Varjo DataStream API can be used to access eye camera video stream in real time.

Following constant specifies the stream type that is reserved for eye camera data stream

/**
 * Distorted (i.e. uncorrected) luminance data stream from eye tracking monochrome camera.
 *
 * Frames contain the following additional data for each channel:
 * - Buffers (access with #varjo_GetBufferId)
 */
static const varjo_StreamType varjo_StreamType_EyeCamera = 3;

When a frame from the eye camera data stream is received, its varjo_StreamFrameMetadata will contain metadata defined by following structure.

/**
 * Metadata for eye camera stream.
 *
 * This metadata is available in #varjo_StreamFrameMetadata structure
 * when #varjo_StreamFrame::type equals #varjo_StreamType_EyeCamera.
 *
 * Glint LEDs are numbered in clockwise direction for left eye and counter clockwise for right
 * eye starting from middle nose side glint LED. Refer to developer documentation for more
 * information.
 */
struct varjo_EyeCameraFrameMetadata {
    varjo_Nanoseconds timestamp;  //!< Timestamp when the frame was captured (start of frame).
    uint32_t glintMaskLeft;       //!< Bit mask of enabled glint LEDs of left chamber
    uint32_t glintMaskRight;      //!< Bit mask of enabled glint LEDs of right chamber
};

See “DataStream API documentation” and ”EyeCameraStreamExample” for more information.

Calibration Options

There are three types of calibration available in the Varjo eye tracking system: quick one-dot Foveation calibration, five-dot Fast and ten-dot Legacy.

One-dot quick Foveation calibration is supported on headsets that utilize foveated rendering - XR-3, VR-3 and Aero. There are several modes available in Varjo Base user interface to calibrate foveated rendering that impact how one-dot Foveation eye tracking calibration is launched:

  • in Always calibrate mode one-dot calibration is launched automatically every time user puts on the headset
  • in Remember my calibration mode one-dot calibration is launched only when user puts on the headset first time and then it is saved and reused in all the next sessions
  • one-dot calibration is not launched automatically in Best estimation without calibration and Never calibrate modes

One-dot Foveation calibration can be also triggered manually from the Varjo Base menu via Calibrate button. After successful one-dot calibration, eye tracking data is immediately available via the API functions retrieving gaze data.

The gaze calibration can be triggered manually with the varjo_RequestGazeCalibrationWithParameters function. Choose between Fast (default), Legacy and OneDot calibration by filling the GazeCalibrationType parameter appropriately. Calling varjo_RequestGazeCalibration will trigger Fast calibration.

Fast calibration is a quick five-dot calibration that uses observations from the current user as well as statistical priors generated from diverse, previously collected and processed observations from other people.

Legacy calibration takes longer and only uses observations from the current user to train the eye tracking algorithm. Legacy calibration can be appropriate, for example, in medical research use cases where the use of statistical priors may not be advisable.

OneDot calibration is the quick calibration that is also used when headset is automatically calibrated for foveated rendering. Like Fast calibration OneDot uses observations from the current user and statistical priors to construct calibration with adequate accuracy. This calibration mode is available only for XR-3, VR-3 and Aero devices.

Note, requesting gaze calibration when headset is not connected will result in error varjo_Error_GazeNotConnected.

Property interface for eye tracking

Following eye tracking related properties can be accessed using varjo_GetPropertyBool and varjo_GetPropertyDouble functions.

  • bool varjo_PropertyKey_UserPresence - Is user wearing the HMD.
  • bool varjo_PropertyKey_GazeCalibrating - Is system currently calibrating the gaze tracker.
  • bool varjo_PropertyKey_GazeCalibrated - Is the gaze tracker calibrated.
  • bool varjo_PropertyKey_GazeAllowed - Is application allowed to use the HMD gaze tracker data.
  • double varjo_PropertyKey_GazeCalibrationQuality_Left - Quality assessment of the left eye gaze calibration. In range [0.0-1.0].
  • double varjo_PropertyKey_GazeCalibrationQuality_Right - Quality assessment of the right eye gaze calibration. In range [0.0-1.0].
  • double varjo_PropertyKey_GazeIPDEstimate - Estimate of user’s interpupillary distance in millimeters.

Example how to use property interface:

// Sync properties from runtime. Call this e.g. once per frame.
varjo_SyncProperties(session);

// Now access properties
bool gazeAllowed = varjo_GetPropertyBool(session, varjo_PropertyKey_GazeAllowed);
double userIpd = varjo_GetPropertyDouble(session, varjo_PropertyKey_GazeIPDEstimate);

Privacy Settings

Varjo Base user interface has privacy setting toggle “Allow eye tracking“, which impacts behaviour of Varjo eye tracking native API.

In case “Allow eye tracking” is off, calling varjo_RequestGazeCalibration, varjo_RequestGazeCalibrationWithParameters or any function retrieving new gaze data will result in error varjo_Error_GazeNotAllowed.

It is possible to query state of this Base setting via varjo_IsGazeAllowed function.

Please note that this privacy setting does not prevent Varjo system to launch quick one-dot Foveation calibration and use eye tracking for foveated rendering.

Examples

Please check ”GazeTrackingExample” application, that demonstrates usage of Varjo native API for eye tracking, and ”EyeCameraStreamExample” application, that demonstrates how to access eye camera stream using native Varjo datastream API.