Foveated Rendering API

Note: you are currently viewing documentation for a beta or an older version of Varjo

Foveated rendering makes use of the eye tracking functionality in Varjo headsets to reduce image quality in peripheral areas, thereby decreasing the rendering workload.

Foveation allows applications to render fewer pixels and achieve a better VR experience. Two foveation modes are supported by the Varjo SDK:

  1. Dynamic viewports
  2. Variable-rate shading (VRS)

Dynamic viewports

When dynamic viewports mode is enabled and gaze information is available, the application can render into a texture that is three times smaller. As an example, for the VR-1 device the optimal texture size is 4096x3200 pixels.

To achieve similar quality in dynamic viewports mode, a texture at 2048x2048 pixels would be sufficient.

Applications should be prepared to render a frame into high-resolution and lower resolution swapchains.

There is no need to create a separate swapchain for the dynamic viewports rendering mode, since the existing swapchain can be used with the help of viewports. However, a new set of textures needs to be created if the engine does not allow you to set the width and height of the render target and always renders into the full texture.

Application flow

  1. The easiest way is to create one big atlas texture at 4096x3200 pixels. This texture can serve both normal and foveated rendering.

  2. Create two sets of viewports for the atlas texture (see the calculateFoveatedViewports() function for an example). Get texture sizes via the new functions varjo_GetTextureSize() and varjo_TextureSize_Type_DynamicFoveation. It also is possible to use different sizes, either larger or smaller.

  3. Enable gaze with varjo_GazeInit(). There is no need to have the user calibrate eye tracking at this point. The application can continue without calibration, but foveation will not be active.

  4. Enable foveation mode with varjo_SetFoveationMode(varjo_DynamicViewport_Mode, varjo_True).

  5. In the render loop, poll events with varjo_PollEvent and check that varjo_EventFoveationStatus is enabled.

  6. Create a projection matrix from field-of-view angles. Important: The matrix must be recreated for every frame since the angles will change depending on gaze.

  7. If foveation is enabled, use foveated viewports to render the frame. Use the normal viewports otherwise.

Example: How to create viewports for the atlas texture

std::vector<varjo_Viewport> calculateFoveatedViewports(varjo_Session* session) {
    const int32_t viewCount = varjo_GetViewCount(session);
    std::vector<varjo_Viewport> viewports;
    viewports.reserve(viewCount);
    int x = 0, y = 0;
    for (int32_t i = 0; i < viewCount; i++) {
        int32_t width = 0, height = 0;
        varjo_GetTextureSize(session, varjo_TextureSize_Type_DynamicFoveation, i, &width, &height);
        const varjo_Viewport viewport = varjo_Viewport{x, y, width, height};
        viewports.push_back(viewport);
        x += viewport.width;
        if (i > 0 && viewports.size() % 2 == 0) {
            x = 0;
            y += viewport.height;
        }
    }

    return viewports;
}

Example: How to integrate dynamic viewport foveating mode into an existing application

...
varjo_GazeInit(session);
varjo_SetFoveationMode(session, varjo_FoveationMode_DynamicFocus, varjo_True);
...
bool useFoveatedViewports = false;
// Render loop starts
while(true) {
    while(varjo_PollEvent(session, &evt)) {
        switch (evt.header.type) {
            ...
            case varjo_EventType_FoveationStatus: {
                        useFoveatedViewports = evt.data.foveationStatus.status == varjo_FoveationStatus_Ok;
                    }
        }
    }
    ...
    varjo_WaitSync(session, frameInfo);
    // Use projection matrix from frameInfo or reconstruct using fov angles (see varjo_GetAlignedView())
    for (int32_t viewIndex = 0; i < varjo_GetViewCount(session); i++) {
        varjo_Viewport viewport;
        if (useFoveatedViewports) {
            viewport = ..... // select foveated viewport
        } else {
            viewport = ..... // select normal viewport
        }
        renderView(viewport, projectionMatrix);
    }
    ...
    // Submit frame as before, making sure that submitInfo contains projectionMatrix which was used to render view
}

Variable-rate shading

Another technique to improve frame rate is variable-rate shading (VRS). At the moment this feature is available for DirectX 12 clients only. For details about VRS, please check the MSDN documentation.

The Varjo SDK provides a function to generate a VRS map:

void varjo_D3D12UpdateVariableRateShadingTexture(
        struct varjo_Session* session,
        struct ID3D12GraphicsCommandList* commandList,
        struct ID3D12Resource* texture,
        enum D3D12_RESOURCE_STATES textureState,
        struct varjo_VariableRateShadingConfig* config);

commandList – It is important to execute this list as late as possible, since the Varjo runtime will continue to update gaze data to the shader even after this function returns in order to decrease latency.

texture – The texture to be used as VRS map.

textureState – Specifies the state of a given texture. The texture will be returned in the same state.

config – Configuration for generating a VRS map.

There are three flags which control how the VRS map will be generated. These can be combined bitwise.

varjo_VariableRateShadingFlag_Stereo – Generates a map for stereo mode (the default is quad mode).

The following picture shows how this can be beneficial:

To generate a high-resolution image inside the focus area we need to render four times the number of pixels in the context area, which will negatively impact performance. However, with the help of VRS we can set a 2x2 shading rate, which means that the GPU will render only one pixel from a 2x2 square.

varjo_VariableRateShadingFlag_Gaze – Generates a map with gaze information if available. Varjo runtime will check for calibrated gaze data and use that to render high resolution only inside the circle. The size of the circle is defined by innerRadius and outerRadius. Tiles outside outerRadius will receive the coarsest possible shading rate (4x4).

varjo_VariableRateShadingFlag_OcclusionMap – Fills pixels that are not visible in the HMD with the coarsest shading rate (4x4).

Examples

Here are several examples of what VRS looks like when overlaid on top of the render target.

  • Blue 1x1
  • Green 2x2
  • Red 4x4

VRS without gaze. Corners will be rendered with the coarsest shading rate since they are not visible in the HMD. Since gaze data is not available, we must render using 1x1 shading rate everywhere else.

VRS with gaze. Almost the entire picture is “red” because the user is looking slightly left and upward, and only that portion of the frame is rendered at full resolution. The outer edge of the circle (green) is rendered with a 2x2 shading rate.

Stereo rendering mode can also benefit from VRS. Since the texture is double the size of the context texture in quad rendering mode, there is no need to render the context area in full resolution.

Stereo rendering mode will benefit even more when gaze is enabled. Only a very small area needs to be rendered in full resolution (part of the focus area).

Implementation

VRS support should be checked first:

bool isVariableRateShadingSupported(ID3D12Device* device)
{
    D3D12_FEATURE_DATA_D3D12_OPTIONS6 options;
    return SUCCEEDED(device->CheckFeatureSupport(D3D12_FEATURE_D3D12_OPTIONS6, &options, sizeof(options)))
        && options.VariableShadingRateTier == D3D12_VARIABLE_SHADING_RATE_TIER_2;
}

The VRS texture has smaller dimensions than the render target. The exact size can be calculated after checking the tile size:

UINT getVariableRateShadingTileSize(ID3D12Device* device)
{
    D3D12_FEATURE_DATA_D3D12_OPTIONS6 options{};
    device->CheckFeatureSupport(D3D12_FEATURE_D3D12_OPTIONS6, &options, sizeof(options));
    return options.ShadingRateImageTileSize;
}

void getVariableRateShadingTextureSize(ID3D12Device* device, UINT textureWidth, UINT textureHeight, UINT* vrsTextureWidth, UINT* vrsTextureHeight)
{
    const UINT tileSize = getVariableRateShadingTileSize(device);
    assert(tileSize);

    *vrsTextureWidth = static_cast<UINT>(ceil(static_cast<float>(textureWidth) / tileSize));
    *vrsTextureHeight = static_cast<UINT>(ceil(static_cast<float>(textureHeight) / tileSize));
}

Another function which can help later is mapping varjo_Viewport to the VRS texture, since it is smaller in size:

varjo_Viewport mapToVrsTexture(const varjo_Viewport& viewport, UINT vrsTileSize)
{
    return varjo_Viewport{static_cast<int32_t>(viewport.x / vrsTileSize), static_cast<int32_t>(viewport.y / vrsTileSize),
        static_cast<int32_t>(viewport.width / vrsTileSize), static_cast<int32_t>(viewport.height / vrsTileSize)};
}

The VRS texture needs to be in the following format: DXGI_FORMAT_R8_UINT and D3D12_RESOURCE_FLAG_ALLOW_UNORDERED_ACCESS flag. It is worth setting the initial state to D3D12_RESOURCE_STATE_SHADING_RATE_SOURCE.

The following function updates the VRS map and command list accordingly (please check the Benchmark project for the complete source code, especially D3D12Renderer.cpp):

void D3D12Renderer::updateVrsMap(ID3D12GraphicsCommandList5* commandList5, const varjo_Viewport& viewport)
{
    const std::shared_ptr<D3D12RenderTarget> dxRenderTarget = std::static_pointer_cast<D3D12RenderTarget, RenderTarget>(m_currentRenderTarget);
    const uint32_t nodeIndex = gpuNodeForView(m_currentViewIndex)->index();
    const varjo_Viewport viewportInVrsTexture = mapToVrsTexture(viewport, m_vrsTileSize);

    const varjo_VariableRateShadingFlags stereoFlag = m_settings.useStereo() ? varjo_VariableRateShadingFlag_Stereo : varjo_VariableRateShadingFlag_None;
    const varjo_VariableRateShadingFlags gazeFlag = m_settings.useGaze() ? varjo_VariableRateShadingFlag_Gaze : varjo_VariableRateShadingFlag_None;

    varjo_VariableRateShadingConfig config{};
    config.viewIndex = m_currentViewIndex;
    config.viewport = viewportInVrsTexture;
    config.flags = varjo_VariableRateShadingFlag_OcclusionMap | stereoFlag | gazeFlag;
    config.innerRadius = 0.1f;
    config.outerRadius = 0.15f;

    varjo_D3D12UpdateVariableRateShadingTexture(m_session, commandList5, *dxRenderTarget->dxVrsTexture(nodeIndex), D3D12_RESOURCE_STATE_SHADING_RATE_SOURCE, &config);

    D3D12_SHADING_RATE_COMBINER chooseScreenspaceImage[2] = {
        D3D12_SHADING_RATE_COMBINER_PASSTHROUGH, D3D12_SHADING_RATE_COMBINER_OVERRIDE};  // Choose screenspace image
    commandList5->RSSetShadingRate(D3D12_SHADING_RATE_4X4, chooseScreenspaceImage);
    commandList5->RSSetShadingRateImage(*dxRenderTarget->dxVrsTexture(nodeIndex));
}