Frame Rate and Latency

The grab rate, set with the camera_fps parameter, is the frame rate at which the camera captures images. The available values depend on the camera model and on the selected resolution, and the grab rate directly affects three aspects of your application:

• Latency: the capture pipeline has a fixed latency measured in frames, so a higher frame rate reduces the time between the moment a scene is captured and the moment your application receives the image.
• Image quality: the exposure time cannot exceed the frame period. A higher frame rate allows shorter exposure times, which reduce motion blur on fast-moving scenes.
• Computation load: by default, the ZED SDK processes every captured frame, so a higher frame rate increases the CPU and GPU load of the host.

This page explains how to select the grab rate for your camera model, and how to combine the highest frame rate with the grab_compute_capping_fps parameter to get the lowest latency without overloading the host.

Available frame rates by camera model

The available frame rates depend on the camera model and on the video mode:

A virtual stereo camera built with two ZED X One cameras uses the frame rates of the ZED X One model it is made of.

Set the grab rate

Select the resolution and the frame rate with the camera_resolution and camera_fps initialization parameters, before opening the camera:

sl::Camera zed;
sl::InitParameters init_params;
init_params.camera_resolution = sl::RESOLUTION::HD1200;
init_params.camera_fps = 60;
sl::ERROR_CODE err = zed.open(init_params);
if (err != sl::ERROR_CODE::SUCCESS)
    exit(-1);

Keep in mind the following rules:

• If camera_fps is set to 0, the default value, the ZED SDK uses the highest frame rate available for the selected resolution.
• If the requested frame rate is not available for the selected resolution, the ZED SDK uses the closest available frame rate.
• If camera_resolution is left at AUTO, the default value, the ZED SDK selects HD1200 for the ZED X and ZED X Mini, and HD720 for the other cameras.
• The resolution and the frame rate cannot be changed while the camera is open. Close and reopen the camera to change them.

To check the frame rate actually reached by your application, call getCurrentFPS() (get_current_fps() in Python) while grabbing. It returns the rate at which grab() succeeds, computed from the camera timestamps of two consecutive successful calls.

Host limits on the grab rate

The frame rate that your system can sustain can be lower than the maximum frame rate of the camera:

• USB cameras: the USB 3.0 bandwidth of a controller is shared by all the cameras connected to it. When the bandwidth is exceeded, corrupted frames appear and cameras can disconnect.
• GMSL2 cameras: each camera uses a dedicated link of the ZED Link capture card or of the ZED Box, but the number of cameras, the resolutions, and the frame rates are limited by the capture card and by the Jetson™ module.
• Computation: if the processing of each frame (depth, positional tracking, object detection, and so on) takes longer than the frame period, grab() runs slower than camera_fps.

The Multi-Camera page gives hardware recommendations for USB and GMSL2 setups.

Minimize the glass-to-glass latency

The glass-to-glass latency is the time between the moment a scene is captured by the camera sensor and the moment the corresponding image is displayed or used by your application. The part introduced by the camera and the ZED SDK capture pipeline is fixed when measured in frames:

• about 2 to 3 frames for GMSL2 cameras,
• about 3 to 4 frames for USB cameras.

This latency is therefore proportional to the frame period: the higher the grab rate, the lower the latency. To minimize it, set camera_fps to the highest frame rate offered by the selected resolution that your host can sustain, as described in Host limits on the grab rate. If latency matters more than image resolution, also consider a lower resolution with a higher maximum frame rate, for example SVGA at 120 fps on a ZED X.

The following table gives the expected latency of the grab operation at the highest frame rate of each video mode:

Reduce the computation load with grab_compute_capping_fps

By default, running the camera at its highest frame rate also forces the ZED SDK to process every frame at that rate, which maximizes the CPU and GPU load. If your application does not need to process all the frames, or if the host cannot sustain this load, use the grab_compute_capping_fps initialization parameter.

grab_compute_capping_fps sets an upper limit to the processing rate of grab(), independently of the capture frame rate:

• The camera keeps capturing at camera_fps, so the capture pipeline latency stays as low as the camera frame rate allows.
• grab() always returns the latest available image, so processing at a lower rate does not make the images older: each processed image is the most recent one captured.
• The ZED SDK modules (image and depth retrieval, positional tracking, object detection, body tracking, and so on) run at most at the capping rate, which reduces the CPU and GPU load.

Unlike camera_fps, the capping value is not limited to the fixed frame rates of the camera: you can set any strictly positive value lower than camera_fps, for example 25.0. Set it to 0 to disable the capping, which is the default behavior.

For example, to capture at 120 fps with a ZED X in SVGA mode for the lowest latency, while processing at most 30 frames per second:

sl::InitParameters init_params;
init_params.camera_resolution = sl::RESOLUTION::SVGA;
init_params.camera_fps = 120;               // Highest frame rate: lowest capture latency
init_params.grab_compute_capping_fps = 30;  // Process at most 30 frames per second

Find the best capping value

The best value depends on the processing power of the host and on the configuration of your application: resolution, depth mode, and enabled modules. To find it:

1. Set camera_fps to the highest frame rate available for your resolution.
2. Set grab_compute_capping_fps to the processing rate required by your application.
3. Run your application with its full workload and monitor the grab rate with getCurrentFPS(), and the CPU and GPU load of the host.
4. If the grab rate stays steadily at the capping value and the host has resources left, you can increase the capping value. If the grab rate cannot reach the capping value, the host cannot sustain this processing rate: decrease the capping value, or reduce the workload, for example with a lighter depth mode.

Tune the power mode of a ZED Box

On a ZED Box Orin or a ZED Box Mini, the power mode of the NVIDIA® Jetson™ module sets the number of active CPU cores and the maximum CPU and GPU frequencies, and therefore the processing rate that the host can sustain. Once your capping value is defined, you can tune the power mode to balance processing frequency and power consumption:

• If the grab rate cannot reach the capping value, select a higher power mode to increase the processing power of the module, within the limits of your power supply.
• If the grab rate stays steadily at the capping value, try lower power modes and keep the lowest one that still sustains the required grab rate. This minimizes the power consumption and the heat of the host.

The available power modes and how to change them are described in the Power Modes guides of the ZED Box Orin and of the ZED Box Mini.