DataStream API

DATASTREAM API

Varjo native datastream API is defined in varjo_datastream.h header. API can be used to access different video streams from a user application. Currently streaming is supported for mixed reality color cameras, HDR cubemap texture and infrared eye cameras. SDK’s Common/DataStreamer class contains an example how this API can be interfaced.

AVAILABLE DATASTREAMS

Stream	Format	Samplerate	Channels	Resolution	Supported devices
Video see through ^{(varjo_StreamType_DistortedColor)}	CPU NV12	90Hz	2	1120x832 (XR-4) 1152x1152 (XR-3)	XR-4, XR-3
HDR environment cubemap ^{(varjo_StreamType_EnvironmentCubemap)}	CPU RGBA16 FLOAT	90Hz	1	256x256(x6)	XR-4, XR-3
Eye tracking ^{(varjo_StreamType_EyeCamera)}	CPU Y8	200Hz	2	640x480 (XR-4) 640x400 (XR-3, VR-3, Aero)	All

USING THE DATASTREAM API

Obtaining available streams

std::vector<varjo_StreamConfig> configs;
configs.resize(varjo_GetDataStreamConfigCount(m_session));
varjo_GetDataStreamConfigs(m_session, configs.data(), static_cast<int32_t>(configs.size()));

Starting a stream

varjo_StartDataStream(m_session, conf.streamId, varjo_ChannelFlag_Left | varjo_ChannelFlag_Right, dataStreamFrameCallback, this);

Stopping a stream

varjo_StopDataStream(m_session, conf.streamId);

Process the frames in the callback functions. Callback contains buffers for both left and right channels as a batch. Note that callback is executed in data stream worker thread.

// "C" trampoline callback function to get back to "C++"
void DataStreamer::dataStreamFrameCallback(const varjo_StreamFrame* frame, varjo_Session* session, void* userData)
{
    DataStreamer* streamer = reinterpret_cast<DataStreamer*>(userData);
    streamer->onDataStreamFrame(frame, session);
}

void DataStreamer::onDataStreamFrame(const varjo_StreamFrame* frame, varjo_Session* session)
{
    // Get frame world pose
    const varjo_Matrix& pose = frame->hmdPose;
    ...

    // Get extra metadata specific for each stream type
    switch (frame->type) {
        case varjo_StreamType_DistortedColor: {
            const varjo_DistortedColorFrameMetadata& frameMetadata = frame->metadata.distortedColor;
            ...
        } break;
        case varjo_StreamType_EnvironmentCubemap: {
            const varjo_EnvironmentCubemapFrameMetadata& frameMetadata = frame->metadata.environmentCubemap;
            ...
        } break;
        case varjo_StreamType_EyeCamera: {
            const varjo_EyeCameraFrameMetadata& frameMetadata = frame->metadata.eyeCamera;
            ...
        } break;
    }

    for (varjo_ChannelIndex channelIndex : {varjo_ChannelIndex_Left, varjo_ChannelIndex_Right}) {
        if (!(frame->channels & (1ull << channelIndex))) {
            continue;
        }

        // Get camera extrinsics if available
        if (frame->dataFlags & varjo_DataFlag_Extrinsics) {
            varjo_Matrix extrinsics = varjo_GetCameraExtrinsics(session, frame->id, frame->frameNumber, channelIndex);
            ...
        }

        // Get camera intrinsics if available
        if (frame->dataFlags & varjo_DataFlag_Intrinsics) {
            varjo_CameraIntrinsics intrinsics = varjo_GetCameraIntrinsics(session, frame->id, frame->frameNumber, channelIndex);
            ...
        }

        // Get buffer ID for the channel, if available
        if (frame->dataFlags & varjo_DataFlag_Buffer) {
            varjo_BufferId bufferId = varjo_GetBufferId(session, frame->id, frame->frameNumber, channelIndex);
            ...
        }
    }
}

Once you have buffer ID, you can access its metadata and buffer contents

// Lock buffer
varjo_LockDataStreamBuffer(session, bufferId);

// Get buffer metadata
varjo_BufferMetadata meta = varjo_GetBufferMetadata(session, bufferId);

// Get buffer contents
if (meta.type == varjo_BufferType_CPU) {
    void* cpuData = varjo_GetBufferCPUData(session, bufferId);
    ...
} else if (meta.type == varjo_BufferType_GPU) {
    varjo_Texture textureId = varjo_GetBufferGPUData(session, bufferId);
    ...
}

// Unlock buffer
varjo_UnlockDataStreamBuffer(session, bufferId);

It is also possible to use delayed buffer handling where buffer is locked in the callback thread, put in queue and then released in render thread.

void DataStreamer::onDataStreamFrame(const varjo_StreamFrame* frame, varjo_Session* session)
{
    ...
    {
        // Lock buffer
        varjo_LockDataStreamBuffer(session, bufferId);

        // Put buffer to queue
        std::lock_guard<std::mutex> lock(m_mutex);
        m_buffers.push_back(bufferId);
    }
    ...
}

void Application::onRenderFrame()
{
    // Pop frames from the queue
    std::vector<varjo_BufferId> buffers;
    {
         std::lock_guard<std::mutex> lock(m_mutex);
         std::swap(buffers, m_buffers);
    }
    
    for (auto bufferId : buffers) {
        // Do something with the buffer
        ...

        // Unlock buffer
        varjo_UnlockDataStreamBuffer(m_session, bufferId);
    }
}