Setting Up Multiple 3D Cameras

There are two ways to build a multi-camera setup:

Connecting multiple cameras on a single machine, such as a desktop PC, server or NVIDIA® Jetson board.
Turning ZEDs into IP cameras that share and exchange synchronized data on the local network.

Setting up Multiple Cameras on One Host #

It is possible to connect multiple ZED cameras on a single Windows or Linux computer. Check out the Multi-Camera sample on GitHub to see an example of implementation.

Hardware Recommendations #

Use PCIe expansion cards: The ZED in 1080p30 mode generates around 250MB/s of image data. USB 3.0 maximum bandwidth is around 620MB/s, so the number of cameras, resolutions and frame rates you can use on a single machine will be limited by the USB 3.0 controller on the motherboard. When the bandwidth limit is exceeded, corrupted frames (green or purple frames, tearing) can appear. To use multiple ZED at full speed on a single computer, we recommend adding USB 3.0 expansion cards such as Inateck 3.0 PCI-E Expansion Cards.
Increase GPU memory: The ZED SDK needs up to 500MB of GPU memory when using a camera in 2K resolution with ULTRA depth mode. To make sure you don’t run out of GPU memory when using the SDK with multiple cameras, we recommend using Geforce 900 series or above with 6GB of memory.

How to list connected cameras #

The ZED SDK can list the connected cameras using getDeviceList(), and provide their serial number for identification. Once you know which ZED should be opened, you can request a specific serial number with InitParameters.input.setFromSerialNumber(1010).

Setting up Multiple Cameras on a Local Network #

Configure the PTP service to synchronize the devices #

Precision Time Protocol (PTP or IEEE1588), an IEEE standard, is a method to synchronize the clock of multiple devices on an Ethernet network. This is achieved by setting one device as the master clock and having all other devices synchronize and periodically adjust to the master clock. Synchronization is handled automatically by the devices once the master and slave devices are set.

PTP is very important to assure that all the cameras in the local network publish frames with a timestamp referred to the same time source because the timestamp uses the system clock for reference.

To configure the Local Network rig it is necessary to select a device to be the master, it will be the only one (if connected to the internet) that keeps the NTP service active to adjust the system clock according to the world time.

Advantages of PTP #

A major advantage of PTP is the hardware support found in various Network Interface Controllers (NICs) and network switches. This specialized hardware allows PTP to account for delays in message transfer and greatly improves the accuracy of time synchronization. Although it is possible to use non-PTP-enabled hardware components within the network, this will often cause increased jitter or introduce delay asymmetry resulting in synchronization inaccuracies, which add up to more PTP-incompatible components used in path communication. To achieve the best possible accuracy, it is recommended that all network components between PTP clocks be enabled for PTP hardware.

With hardware PTP support, the NIC has its own built-in clock, which is used to time stamp received and transmitted PTP messages. It is this onboard clock that is synchronized with the PTP master and the computer system clock is synchronized with the PTP hardware clock on the NIC. With PTP software support, the system clock is used to time stamp PTP messages and is synchronized directly with the PTP master. Hardware PTP support provides greater accuracy as the NIC can stamp PTP packets as they are sent and received, while software PTP support requires additional processing of PTP packets by the operating system.

Install PTP and tools #

To enable each device of the Local Network to receive PTP messages you must install the linuxptp and the ethtool packages:

$ sudo apt install linuxptp ethtool

The linuxptp package includes the ptp4l and the phc2sys programs for clock synchronization. The ptp4l program implements the PTP limit clock and the ordinary clock. With hardware timestamp, it is used to synchronize the PTP hardware clock with the master clock and with the software timestamp, synchronizing the system clock with the master clock. The phc2sys program is only needed with hardware timestamps, to synchronize the system clock with the PTP hardware clock on the network interface card.

ethtool is used to verify the clock capabilities of the NIC installed on the devices.

Check PTP hardware support #

To verify the capabilities of the NIC installed on the device, use the following command:

$ ethtool -T <interface_name>

The network interface name can be retrieved using the command:

$ nmcli device status

The DEVICE column shows the name of all the network interfaces currently connected.

For software time stamping support, the parameters list should include:

SOF_TIMESTAMPING_SOFTWARE
SOF_TIMESTAMPING_TX_SOFTWARE
SOF_TIMESTAMPING_RX_SOFTWARE

For hardware time stamping support, the parameters list should include:

SOF_TIMESTAMPING_RAW_HARDWARE
SOF_TIMESTAMPING_TX_HARDWARE
SOF_TIMESTAMPING_RX_HARDWARE

For example, the ethernet adapter of the NVIDIA® Jetson TX2 supports hardware and software timestamping:

$ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
	hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
	software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
	hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
	software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
	software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
	hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
	off                   (HWTSTAMP_TX_OFF)
	on                    (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
	none                  (HWTSTAMP_FILTER_NONE)
	ptpv1-l4-sync         (HWTSTAMP_FILTER_PTP_V1_L4_SYNC)
	ptpv1-l4-delay-req    (HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ)
	ptpv2-l4-sync         (HWTSTAMP_FILTER_PTP_V2_L4_SYNC)
	ptpv2-l4-delay-req    (HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ)
	ptpv2-l2-sync         (HWTSTAMP_FILTER_PTP_V2_L2_SYNC)
	ptpv2-l2-delay-req    (HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ)
	ptpv2-event           (HWTSTAMP_FILTER_PTP_V2_EVENT)

while the WiFi adapter cannot be used for PTP synchronization, it does not support hardware timestamping and it does not provide full software timestamping support, because the SOF_TIMESTAMPING_TX_SOFTWARE capability is lacking:

$ ethtool -T wlan0
Time stamping parameters for wlan0:
Capabilities:
	software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
	software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none

Setup the master device #

If the master device is connected to the internet and you want to be sure that the time of all the devices in the local network is synchronized with the world time, enable the NTP sync service:

$ timedatectl set-ntp on

Open a console and start the PTP service in hardware mode:

$ sudo ptp4l -i <interface_name> -m

Replace <interface_name> with the name of the network interface that you are using for communication (e.g. eth0, wlan0, eno1). You can get the list of available network devices using the command

$ nmcli device status

The DEVICE column shows the name of all the network interfaces currently connected.

If the ptp4l command reports an error message similar to the following

ptp4l[861.317]: interface '<interface_name>' does not support requested timestamping mode
failed to create a clock

then the selected network interface does not support the hardware timestamping and the PTP service must be started in “software mode”:

$ sudo ptp4l -i <interface_name> -S -m

Note: Remove the -m option to stop console logging messages.

Setup all the slave devices #

Please replicate the following procedure for all the slave devices to be synchronized on the Local Network.

Open a console and enter the following command to start the PTP service in slave mode:

$ sudo ptp4l -i <interface_name> -s -m

Replace <interface_name> with the name of the network interface that you are using for communication (e.g. eth0, wlan0, eno1).

Like for the master, if the ptp4l command reports an error message similar to the following

ptp4l[861.317]: interface '<interface_name>' does not support requested timestamping mode
failed to create a clock

then the network interface does not support the hardware timestamping and the PTP service must be started in “software mode”:

$ sudo ptp4l -i <interface_name> -S -s -m

Note: Remove the -m option to stop console logging messages.

If the slave has been started in hardware timestamping mode, it is necessary to synchronize the system clock to the PTP hardware clock on the NIC. Open a new console and enter the command:

$ sudo phc2sys -m -s /dev/ptp0 -c CLOCK_REALTIME -O 0

The -s option sets the clock source, in this case, it is the /dev/ptp0 created above with the ptp4l command.

The -c option sets the destination clock to be synchronized, in this case, the real-time clock of the operating system.

It is important to explicitly set the “clock offset” to zero with the option -O 0, otherwise a default offset of ~37 seconds will be applied (see the TIME SCALE USAGE section of the phc2sys man page with $ man phc2sys).

Note: Remove the -m option to stop console logging messages.

Test the configuration #

To test the correctness of the configuration you must disable the NTP service also on the master to be able to manually change the system time:

$ sudo timedatectl set-ntp off

Compile the following simple ZED application:

// Standard includes
#include <stdio.h>
#include <string.h>

// ZED includes
#include <sl/Camera.hpp>

// Using namespace
using namespace sl;
using namespace std;

int main(int argc, char **argv) {
    // Create a ZED camera
    Camera zed;

    // Set configuration parameters for the ZED
    InitParameters init_parameters;
    init_parameters.camera_resolution = RESOLUTION::HD720;
    init_parameters.depth_mode = DEPTH_MODE::NONE;
    init_parameters.sdk_verbose = true;
    int res_arg = parseArgs(argc, argv, init_parameters);

    // Open the camera
    auto returned_state = zed.open(init_parameters);
    if (returned_state != ERROR_CODE::SUCCESS) {
        print("Camera Open", returned_state, "Exit program.");
        return EXIT_FAILURE;
    }

    while (1) {
        if (zed.grab() == ERROR_CODE::SUCCESS)
        {
            sl::Timestamp image_ts = zed.getTimestamp(sl::TIME_REFERENCE::IMAGE);
            sl::Timestamp current_ts = zed.getTimestamp(sl::TIME_REFERENCE::CURRENT);

            long long diff_ms = (long long)current_ts.getMilliseconds() - (long long)image_ts.getMilliseconds();
            std::cout << " Latency/Diff (ms) " << diff_ms << std::endl;
        }
        else
            sleep_ms(1);
    }

    // disable Streaming
    zed.disableStreaming();

    // close the Camera
    zed.close();
    return EXIT_SUCCESS;
}

The application compares the timestamps between the CURRENT reference and the IMAGE reference. In normal conditions the value printed while the application runs should stay quite constant, the value depends on the framerate:

Latency/Diff (ms) <value0>
Latency/Diff (ms) <value1>
Latency/Diff (ms) <value2>
.
.
.
Latency/Diff (ms) <valueN>

However, if a clock change occurs due to PTP sync:

The CURRENT timestamp will change accordingly to the clock configuration change (smooth or jump).
The IMAGE timestamp will be smoothly adapted to the current system clock. There will be no big jumps because it needs to stay consistent with the camera FPS. Therefore, you should see an initial big difference that decreases over time to finally go back to the original constant difference.

To generate a visible PTP Sync you can manually modify the system time of the master device:

$ sudo date --set="2021-01-20 15:30:00.000"

The greater the difference between the new time and the previous one, the greater the time jump highlighted by the test application, and the greater the time taken to stabilize.

Note: do not forget to re-enable the NTP service on the master device after the test to guarantee that the system clocks are synchronized with the world time:

sudo timedatectl set-ntp on