DataStream API
Note: you are currently viewing documentation for a beta or an older version of Varjo
DATASTREAM API
Varjo native datastream API is defined in varjo_datastream.h
header. API can be used to access different video streams from a user application. Currently streaming is supported for mixed reality color cameras, HDR cubemap texture and infrared eye cameras. SDK’s Common/DataStreamer
class contains an example how this API can be interfaced.
AVAILABLE DATASTREAMS
Stream | Format | Samplerate | Channels | Resolution | Supported devices |
---|---|---|---|---|---|
Video see through (varjo_StreamType_DistortedColor) |
CPU NV12 | 90Hz | 2 | 1152x1152 | XR-3 |
Video see through (varjo_StreamType_DistortedColor) |
CPU YUV422 semiplanar |
90Hz | 2 | 1008x1008 | XR-1 |
HDR environment cubemap (varjo_StreamType_EnvironmentCubemap) |
CPU RGBA16 FLOAT | 90Hz | 1 | 6x(256x256) | XR-3, XR-1 |
Eye tracking (varjo_StreamType_EyeCamera) |
CPU Y8 | 200Hz | 2 | 640x400 | XR-3, VR-3, Aero |
USING THE DATASTREAM API
Obtaining available streams
std::vector<varjo_StreamConfig> configs;
configs.resize(varjo_GetDataStreamConfigCount(m_session));
varjo_GetDataStreamConfigs(m_session, configs.data(), static_cast<int32_t>(configs.size()));
Starting a stream
varjo_StartDataStream(m_session, conf.streamId, varjo_ChannelFlag_Left | varjo_ChannelFlag_Right, dataStreamFrameCallback, this);
Stopping a stream
varjo_StopDataStream(m_session, conf.streamId);
Process the frames in the callback functions. Callback contains buffers for both left and right channels as a batch. Note that callback is executed in data stream worker thread.
// "C" trampoline callback function to get back to "C++"
void DataStreamer::dataStreamFrameCallback(const varjo_StreamFrame* frame, varjo_Session* session, void* userData)
{
DataStreamer* streamer = reinterpret_cast<DataStreamer*>(userData);
streamer->onDataStreamFrame(frame, session);
}
void DataStreamer::onDataStreamFrame(const varjo_StreamFrame* frame, varjo_Session* session)
{
// Get frame world pose
const varjo_Matrix& pose = frame->hmdPose;
...
// Get extra metadata specific for each stream type
switch (frame->type) {
case varjo_StreamType_DistortedColor: {
const varjo_DistortedColorFrameMetadata& frameMetadata = frame->metadata.distortedColor;
...
} break;
case varjo_StreamType_EnvironmentCubemap: {
const varjo_EnvironmentCubemapFrameMetadata& frameMetadata = frame->metadata.environmentCubemap;
...
} break;
case varjo_StreamType_EyeCamera: {
const varjo_EyeCameraFrameMetadata& frameMetadata = frame->metadata.eyeCamera;
...
} break;
}
for (varjo_ChannelIndex channelIndex : {varjo_ChannelIndex_Left, varjo_ChannelIndex_Right}) {
if (!(frame->channels & (1ull << channelIndex))) {
continue;
}
// Get camera extrinsics if available
if (frame->dataFlags & varjo_DataFlag_Extrinsics) {
varjo_Matrix extrinsics = varjo_GetCameraExtrinsics(session, frame->id, frame->frameNumber, channelIndex);
...
}
// Get camera intrinsics if available
if (frame->dataFlags & varjo_DataFlag_Intrinsics) {
varjo_CameraIntrinsics intrinsics = varjo_GetCameraIntrinsics(session, frame->id, frame->frameNumber, channelIndex);
...
}
// Get buffer ID for the channel, if available
if (frame->dataFlags & varjo_DataFlag_Buffer) {
varjo_BufferId bufferId = varjo_GetBufferId(session, frame->id, frame->frameNumber, channelIndex);
...
}
}
}
Once you have buffer ID, you can access its metadata and buffer contents
// Lock buffer
varjo_LockDataStreamBuffer(session, bufferId);
// Get buffer metadata
varjo_BufferMetadata meta = varjo_GetBufferMetadata(session, bufferId);
// Get buffer contents
if (meta.type == varjo_BufferType_CPU) {
void* cpuData = varjo_GetBufferCPUData(session, bufferId);
...
} else if (meta.type == varjo_BufferType_GPU) {
varjo_Texture textureId = varjo_GetBufferGPUData(session, bufferId);
...
}
// Unlock buffer
varjo_UnlockDataStreamBuffer(session, bufferId);
It is also possible to use delayed buffer handling where buffer is locked in the callback thread, put in queue and then released in render thread.
void DataStreamer::onDataStreamFrame(const varjo_StreamFrame* frame, varjo_Session* session)
{
...
{
// Lock buffer
varjo_LockDataStreamBuffer(session, bufferId);
// Put buffer to queue
std::lock_guard<std::mutex> lock(m_mutex);
m_buffers.push_back(bufferId);
}
...
}
void Application::onRenderFrame()
{
// Pop frames from the queue
std::vector<varjo_BufferId> buffers;
{
std::lock_guard<std::mutex> lock(m_mutex);
std::swap(buffers, m_buffers);
}
for (auto bufferId : buffers) {
// Do something with the buffer
...
// Unlock buffer
varjo_UnlockDataStreamBuffer(m_session, bufferId);
}
}