Varjo Markers

Varjo Markers can be used to track static or dynamic real-world objects using the video pass-through cameras on Varjo XR-3 headset.

See Varjo Markers for more general information and printing instructions.

Varjo Markers are typically used in mixed reality applications, but can be also used without video pass-through rendering in virtual reality applications to align virtual objects in the scene with physical objects in the real world.

This document describes how to use Varjo Markers with the Varjo XR Plug-in.

Making a GameObject track a Varjo Marker

Start by printing out a Varjo Marker. Printable markers and instructions can be found on the Varjo Markers page. Please read the printing instructions carefully for optimal printing quality.

In this documentation we use a large marker with ID 350.

You will need a manager script to handle all the tracked markers in your Unity scene.

Let’s create a new script called Varjo Marker Manager and add it in one of the GameObjects in the scene.

Like the other Varjo XR plugin features, Varjo Markers can be found in the Varjo.XR namespace. Make sure to add the using directive at the beginning of the CS file.

using Varjo.XR;

Next you need to add a way to define GameObjects that will be following the tracked markers. For example, you can create a simple array of structs to store ID-GameObject pairs that can be edited in the Unity Inspector. We also include a boolean for controlling the tracking mode of the marker. Add this piece of code in the beginning of the VarjoMarkerManager class.

// Serializable struct to make it easy to add tracked objects in the Inspector. 
[Serializable] 
public struct TrackedObject 
{ 
    public long id; 
    public GameObject gameObject;
    public bool dynamicTracking;
} 

// An public array for all the tracked objects. 
public TrackedObject[] trackedObjects = new TrackedObject[1];

Add a couple more fields to the class, since the manager script needs to keep track of all the markers in the environment. Add one list to store found markers and another one for the IDs of markers that have detected before, but have not been visible for longer than the duration of the marker timeout.

// A list for found markers.
private List<VarjoMarker> markers = new List<VarjoMarker>();

// A list for IDs of removed markers.
private List<long> removedMarkerIds = new List<long>();

Before you try to get marker data, you will need to enable Varjo Marker tracking.

Add the following to have Varjo Markers enabled only when the object is active and enabled.

private void OnEnable()
{
    // Enable Varjo Marker tracking.
    VarjoMarkers.EnableVarjoMarkers(true);
}

private void OnDisable()
{
    // Disable Varjo Marker tracking.
    VarjoMarkers.EnableVarjoMarkers(false);
}

Now all you need is the actual logic of the marker manager. In this example script, we update the marker data in the Update loop. If Varjo Marker tracking is enabled, we get a list of the latest marker data and update all GameObjects in our trackedObjects array which have an ID matching that of a marker in the markers array. We also set the tracking mode of the marker here if required.

Varjo Markers have two tracking modes; Stationary and Dynamic. The default tracking mode is Stationary, which means the pose is heavily filtered and it’s less responsive to marker’s movements. This is optimal, if you want to align fixed physical elements like a cockpit on a flight simulator with the virtual elements. When the tracking mode is Dynamic, the pose updates are more responsive to marker’s movements. This should be used, if the marker is expected to move. Use DoPrediction flag to change the tracking mode. You can enable dynamic tracking with VarjoMarkers.AddVarjoMarkerFlags(marker.id, VarjoMarkerFlags.DoPrediction) and disable it with VarjoMarkers.RemoveVarjoMarkerFlags(marker.id, VarjoMarkerFlags.DoPrediction).

In the Update loop we also get the list of IDs of removed markers and deactivate any matching GameObject in the trackedObjects array.

void Update()
{
    // Check if Varjo Marker tracking is enabled and functional.
    if (VarjoMarkers.IsVarjoMarkersEnabled())
    {
        // Get a list of markers with up-to-date data.
        VarjoMarkers.GetVarjoMarkers(out markers);

        // Loop through found markers and update gameobjects matching the marker ID in the array.
        foreach (var marker in markers)
        {
            for (var i = 0; i < trackedObjects.Length; i++)
            {
                if (trackedObjects[i].id == marker.id)
                {
                    // This simple marker manager controls only visibility and pose of the GameObjects.
                    trackedObjects[i].gameObject.SetActive(true);
                    trackedObjects[i].gameObject.transform.localPosition = marker.pose.position;
                    trackedObjects[i].gameObject.transform.localRotation = marker.pose.rotation;

                    // Set the marker tracking mode
                    if ((marker.flags == VarjoMarkerFlags.DoPrediction) != trackedObjects[i].dynamicTracking)
                        {
                            if (trackedObjects[i].dynamicTracking)
                            {
                                VarjoMarkers.AddVarjoMarkerFlags(marker.id, VarjoMarkerFlags.DoPrediction);
                            }
                            else
                            {
                                VarjoMarkers.RemoveVarjoMarkerFlags(marker.id, VarjoMarkerFlags.DoPrediction);
                            }
                        }
                }
            }
        }

        // Get a list of IDs of removed markers.
        VarjoMarkers.GetRemovedVarjoMarkerIds(out removedMarkerIds);

        // Loop through removed marker IDs and deactivate gameobjects matching the marker IDs in the array.
        foreach (var id in removedMarkerIds)
        {
            for (var i = 0; i < trackedObjects.Length; i++)
            {
                if (trackedObjects[i].id == id)
                {
                    trackedObjects[i].gameObject.SetActive(false);
                }
            }
        }
    }
}

Back in the Unity Editor, select the GameObject you want to follow the Varjo Marker.

As the pose in the Varjo Marker is relative to the tracking space, you probably want to make the GameObject a sibling of the XR camera. Make sure that the GameObject is not flagged as static.

Add the GameObject in the array of TrackedObjects in your VarjoMarkerManager and set the ID matching the ID of a printed Varjo Marker.

If the marker is intended to be moved, enable Dynamic Tracking.

You can have as many tracked objects as you want as long you don’t use multiple printed markers with the same ID in the environment.

Press Play and look around. If the video pass-through cameras on the headset see a printed marker with an ID matching one of the TrackedObjects, you should see the GameObject following the position and rotation of the marker.

A Varjo Marker contains much more than just position and rotation. You can, for example, easily make the size of the object match the size of the Varjo Marker. If you do this, please note that the y-size of a flat marker is very small, so you probably want to use x or z axis to control also the height of the object.

The ID and size of a Varjo Marker never change, so it is enough to set them when the marker is found for the first time.

Using Varjo Markers to align a video pass-through mask with a real-world object

One of the most common uses for Varjo Markers is aligning a video pass-through mask with a physical object in the environment. This is useful when bringing objects such as input devices or instruments from the real world into the virtual environment.

Follow the instructions below to create a simple mask with which to display a keyboard inside the virtual environment.

Continuing from the instructions above, replace a tracked GameObject with an empty GameObject. Add this GameObject in the TrackedObjects array with an ID matching your printed Varjo Marker.

If the keyboard is intended to be moved, enable Dynamic Tracking. When masking stationary objects, it’s better to keep Dynamic Tracking disabled.

Place the printed marker next to your keyboard. Typically you would want to attach the marker to the physical object, so that they move together.

In the Unity Editor, add a GameObject with a mesh for the mask as a child of the tracked GameObject. For the best quality you would want the mesh to be the same shapes as the physical object. For the sake of simplicity, we will use a Cube primitive and simply match the scale of the GameObject with the size of the keyboard.

Measure the offset from the center of the printed Varjo Marker to the center of your keyboard. Set this offset as the local position of your mask.

Follow the instructions in Masking with Varjo XR Plug-in to set up a video pass-through mask in your project.

Assign the VST Mask material to your mask mesh.

Press Play. You should now be able to see the keyboard inside your virFtual environment. If the Varjo Marker is attached to the keyboard, you can move the keyboard and the mask will follow.