Setting Up Multiple 3D Cameras

There are two ways to build a multi-camera setup:

• Connecting multiple cameras on a single machine, such as a desktop PC, server or NVIDIA® Jetson™ board.
• Turning ZEDs into IP cameras that share and exchange synchronized data on the local network.

StereoLabs cameras come in two connectivity families, and several of the recommendations below depend on which one you use:

• USB cameras (ZED, ZED Mini, ZED 2, ZED 2i): connect over USB 3.0 to any Windows or Linux host.
• GMSL2 cameras (ZED X, ZED X Mini, ZED X Nano, ZED X One S/GS, ZED X One 4K): connect through a ZED Link GMSL2 capture card and are only compatible with NVIDIA® Jetson™ platforms.

Each section below indicates whether it applies to USB, GMSL2, or both. Both families can be combined on a single host and can be used as IP cameras on a local network.

Setting up Multiple Cameras on One Host

It is possible to connect multiple ZED cameras to a single host. Check out the Multi-Camera sample on GitHub to see an example of implementation.

The way cameras are connected and the limits you have to consider depend on whether they are USB or GMSL2 cameras.

Hardware Recommendations for USB cameras

Applies to USB cameras (ZED, ZED Mini, ZED 2, ZED 2i).

• Watch the USB bandwidth: The ZED in 1080p30 mode generates around 250MB/s of image data. The maximum USB 3.0 bandwidth is around 620MB/s, so the number of cameras, resolutions, and frame rates you can use on a single machine will be limited by the USB 3.0 controller on the motherboard. When the bandwidth limit is exceeded, corrupted frames (green or purple frames, tearing) can appear, and cameras may even disconnect.

• Use PCIe expansion cards: To use multiple USB ZED cameras at full speed on a single computer, we recommend distributing them across several USB controllers by adding USB 3.0 multi-channel expansion cards.

Hardware Recommendations for GMSL2 cameras

Applies to GMSL2 cameras (ZED X, ZED X Mini, ZED X Nano, ZED X One S/GS, ZED X One 4K).

GMSL2 cameras do not connect over USB. They are linked to an NVIDIA® Jetson™ device through a ZED Link GMSL2 capture card, which provides dedicated, low-latency serial links instead of a shared USB bus. This removes the USB-controller bottleneck and makes GMSL2 the recommended option for dense multi-camera rigs.

• Choose the right capture card: each GMSL2 link drives one camera. Pick the capture card according to the number of cameras you need to connect to a single Jetson™:

  • ZED Link Mono: 1 camera.
  • ZED Link Duo: up to 4 cameras.
  • ZED Link Quad: up to 8 cameras.

• Mind the power budget: connecting several GMSL2 cameras (especially when powering accessories over the GMSL2 link) increases the current draw on the capture card. Check the GMSL2 Power Requirements before scaling up a rig.

• Install the ZED Link driver: GMSL2 cameras require the ZED Link driver to be installed on the Jetson™ before they can be opened by the ZED SDK. See Install and Upgrade the Drivers.

Increase GPU memory

Applies to both USB and GMSL2 cameras.

The ZED SDK needs up to 750MB of GPU memory when retrieving depth information in NEURAL LIGHT mode. To make sure you don’t run out of GPU memory when using the SDK with multiple cameras, we recommend using an NVIDIA® GPU with at least 6GB of memory. On NVIDIA® Jetson™ platforms, where memory is shared between CPU and GPU, make sure the device has enough total RAM for the number of cameras and depth modes you plan to run.

Frame synchronization

GMSL2 cameras can be synchronized at the hardware level for accurate, frame-level alignment, while USB cameras rely on timestamp-based software alignment.

• GMSL2 (hardware triggering): cameras connected to the same ZED Link Duo or Quad capture card are automatically frame-synchronized. The capture card can also act as a trigger master to drive external sensors, or as a slave to be driven by an external trigger source, and multiple capture cards can be chained together so all their cameras share the same trigger. See the GPIO triggering guides for the ZED Link Mono, ZED Link Duo, and ZED Link Quad cards.

• USB: USB cameras do not expose a hardware trigger. To align frames across USB cameras, rely on the image timestamps provided by the ZED SDK, or use the local network setup with PTP synchronization described below.

How to list connected cameras

Applies to both USB and GMSL2 cameras.

The ZED SDK can list the connected cameras using getDeviceList(), and provide their serial number for identification. Once you know which ZED should be opened, you can request a specific serial number with InitParameters.input.setFromSerialNumber(1010).

Setting up Multiple Cameras on a Local Network

Applies to both USB and GMSL2 cameras.

Both USB and GMSL2 cameras can be turned into IP cameras and stream synchronized data over the local network. Because synchronization here relies on the system clock of each host rather than on a physical trigger line, the Precision Time Protocol (PTP) is used to align the clocks of all the devices on the network.

Configure the PTP service to synchronize the devices

Precision Time Protocol (PTP or IEEE1588), an IEEE standard, is a method to synchronize the clock of multiple devices on an Ethernet network. This is achieved by setting one device as the master clock and having all other devices synchronize and periodically adjust to the master clock. Synchronization is handled automatically by the devices once the master and slave devices are set.

PTP is essential to ensure that all the cameras on the local network publish frames with timestamps referenced to the same time source, since each timestamp uses the system clock for reference.

To configure the local network rig, you must select one device to act as the master. It will be the only one (if connected to the internet) that keeps the NTP service active to adjust the system clock according to world time.

Advantages of PTP

A major advantage of PTP is the hardware support found in various Network Interface Controllers (NICs) and network switches. This specialized hardware allows PTP to account for delays in message transfer and greatly improves the accuracy of time synchronization. Although it is possible to use non-PTP-enabled hardware components within the network, doing so often causes increased jitter or introduces delay asymmetry, resulting in synchronization inaccuracies that grow with the number of PTP-incompatible components in the communication path. To achieve the best possible accuracy, it is recommended that all network components between PTP clocks support PTP hardware.

With hardware PTP support, the NIC has its own built-in clock, which is used to time stamp received and transmitted PTP messages. It is this onboard clock that is synchronized with the PTP master, and the computer system clock is synchronized with the PTP hardware clock on the NIC. With PTP software support, the system clock is used to time stamp PTP messages and is synchronized directly with the PTP master. Hardware PTP support provides greater accuracy as the NIC can stamp PTP packets as they are sent and received, while software PTP support requires additional processing of PTP packets by the operating system.

Install PTP and tools

To enable each device on the local network to receive PTP messages, you must install the linuxptp and ethtool packages:

$
sudo apt install linuxptp ethtool

The linuxptp package includes the ptp4l and phc2sys programs for clock synchronization. The ptp4l program implements the PTP boundary clock and the ordinary clock. With hardware timestamping, it synchronizes the PTP hardware clock with the master clock; with software timestamping, it synchronizes the system clock with the master clock. The phc2sys program is only needed with hardware timestamps, to synchronize the system clock with the PTP hardware clock on the network interface card.

ethtool is used to verify the clock capabilities of the NIC installed on the devices.

Check PTP hardware support

To verify the capabilities of the NIC installed on the device, use the following command:

$
ethtool -T <interface_name>

The network interface name can be retrieved using the command:

$
nmcli device status

The DEVICE column shows the name of all the network interfaces currently connected.

For software time stamping support, the parameters list should include:

• SOF_TIMESTAMPING_SOFTWARE
• SOF_TIMESTAMPING_TX_SOFTWARE
• SOF_TIMESTAMPING_RX_SOFTWARE

For hardware time stamping support, the parameters list should include:

• SOF_TIMESTAMPING_RAW_HARDWARE
• SOF_TIMESTAMPING_TX_HARDWARE
• SOF_TIMESTAMPING_RX_HARDWARE

For example, the Ethernet adapter of the NVIDIA® Jetson™ TX2 supports hardware and software timestamping:

$
ethtool -T eth0
$
Time stamping parameters for eth0:
$
Capabilities:
$
    hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
$
    software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
$
    hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
$
    software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
$
    software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
$
    hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
$
PTP Hardware Clock: 0
$
Hardware Transmit Timestamp Modes:
$
    off                   (HWTSTAMP_TX_OFF)
$
    on                    (HWTSTAMP_TX_ON)
$
Hardware Receive Filter Modes:
$
    none                  (HWTSTAMP_FILTER_NONE)
$
    ptpv1-l4-sync         (HWTSTAMP_FILTER_PTP_V1_L4_SYNC)
$
    ptpv1-l4-delay-req    (HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ)
$
    ptpv2-l4-sync         (HWTSTAMP_FILTER_PTP_V2_L4_SYNC)
$
    ptpv2-l4-delay-req    (HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ)
$
    ptpv2-l2-sync         (HWTSTAMP_FILTER_PTP_V2_L2_SYNC)
$
    ptpv2-l2-delay-req    (HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ)
$
    ptpv2-event           (HWTSTAMP_FILTER_PTP_V2_EVENT)

The WiFi adapter, on the other hand, cannot be used for PTP synchronization: it does not support hardware timestamping and does not provide full software timestamping support, because the SOF_TIMESTAMPING_TX_SOFTWARE capability is missing:

$
ethtool -T wlan0
$
Time stamping parameters for wlan0:
$
Capabilities:
$
    software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
$
    software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
$
PTP Hardware Clock: none
$
Hardware Transmit Timestamp Modes: none
$
Hardware Receive Filter Modes: none

Set up the master device

If the master device is connected to the internet and you want to be sure that the time of all the devices in the local network is synchronized with the world time, enable the NTP sync service:

$
timedatectl set-ntp on

Open a console and start the PTP service in hardware mode:

$
sudo ptp4l -i <interface_name> -m

Replace <interface_name> with the name of the network interface that you are using for communication (e.g. eth0, wlan0, eno1). You can get the list of available network devices using the command

$
nmcli device status

The DEVICE column shows the name of all the network interfaces currently connected.

If the ptp4l command reports an error message similar to the following

$
ptp4l[861.317]: interface '<interface_name>' does not support requested timestamping mode
$
failed to create a clock

then the selected network interface does not support hardware timestamping and the PTP service must be started in “software mode”:

$
sudo ptp4l -i <interface_name> -S -m

Set up all the slave devices

Please replicate the following procedure for all the slave devices to be synchronized on the local network.

Open a console and enter the following command to start the PTP service in slave mode:

$
sudo ptp4l -i <interface_name> -s -m 

Replace <interface_name> with the name of the network interface that you are using for communication (e.g. eth0, wlan0, eno1).

As with the master, if the ptp4l command reports an error message similar to the following

$
ptp4l[861.317]: interface '<interface_name>' does not support requested timestamping mode
$
failed to create a clock

then the network interface does not support hardware timestamping and the PTP service must be started in “software mode”:

$
sudo ptp4l -i <interface_name> -S -s -m

If the slave has been started in hardware timestamping mode, it is necessary to synchronize the system clock to the PTP hardware clock on the NIC. Open a new console and enter the command:

$
sudo phc2sys -m -s /dev/ptp0 -c CLOCK_REALTIME -O 0

The -s option sets the clock source, in this case the /dev/ptp0 clock created above with the ptp4l command.

The -c option sets the destination clock to be synchronized, in this case the real-time clock of the operating system.

It is important to explicitly set the “clock offset” to zero with the option -O 0, otherwise a default offset of ~37 seconds will be applied (see the TIME SCALE USAGE section of the phc2sys man page with man phc2sys).

Test the configuration

To test the correctness of the configuration, you must also disable the NTP service on the master so that you can manually change the system time:

$
sudo timedatectl set-ntp off

Compile the following simple ZED application:

1
// Standard includes
2
#include <stdio.h>
3
#include <string.h>
4
5
// ZED includes
6
#include <sl/Camera.hpp>
7
8
// Using namespace
9
using namespace sl;
10
using namespace std;
11
12
int main(int argc, char **argv) {
13
    // Create a ZED camera
14
    Camera zed;
15
16
    // Set configuration parameters for the ZED
17
    InitParameters init_parameters;
18
    init_parameters.camera_resolution = RESOLUTION::HD720;
19
    init_parameters.depth_mode = DEPTH_MODE::NONE;
20
    init_parameters.sdk_verbose = true;
21
    int res_arg = parseArgs(argc, argv, init_parameters);
22
23
    // Open the camera
24
    auto returned_state = zed.open(init_parameters);
25
    if (returned_state != ERROR_CODE::SUCCESS) {
26
        print("Camera Open", returned_state, "Exit program.");
27
        return EXIT_FAILURE;
28
    }
29
30
    while (1) {
31
        if (zed.grab() == ERROR_CODE::SUCCESS)
32
        {
33
            sl::Timestamp image_ts = zed.getTimestamp(sl::TIME_REFERENCE::IMAGE);
34
            sl::Timestamp current_ts = zed.getTimestamp(sl::TIME_REFERENCE::CURRENT);
35
36
            long long diff_ms = (long long)current_ts.getMilliseconds() - (long long)image_ts.getMilliseconds();
37
            std::cout << " Latency/Diff (ms) " << diff_ms << std::endl;
38
        }
39
        else
40
            sleep_ms(1);
41
    }
42
43
    // disable Streaming
44
    zed.disableStreaming();
45
46
    // close the Camera
47
    zed.close();
48
    return EXIT_SUCCESS;
49
}

The application compares the timestamps from the CURRENT reference and the IMAGE reference. Under normal conditions, the value printed while the application runs should stay fairly constant; the value depends on the frame rate:

$
Latency/Diff (ms) <value0>
$
Latency/Diff (ms) <value1>
$
Latency/Diff (ms) <value2>
$
.
$
.
$
.
$
Latency/Diff (ms) <valueN>

However, if a clock change occurs due to PTP sync:

• The CURRENT timestamp will change according to the clock configuration change (smooth or jump).
• The IMAGE timestamp will be smoothly adapted to the current system clock. There will be no large jumps because it needs to stay consistent with the camera FPS. Therefore, you should see an initial large difference that decreases over time and finally returns to the original constant difference.

To generate a visible PTP sync, you can manually modify the system time of the master device:

$
sudo date --set="2021-01-20 15:30:00.000"

The greater the difference between the new time and the previous one, the greater the time jump highlighted by the test application, and the greater the time taken to stabilize.

$
sudo timedatectl set-ntp on