Setting up the ZED X One Stereo

Install the Software #

The ZED X One is compatible with NVIDIA® Jetson platforms and has similar compatibility to the ZED X. You can refer to the hardware and driver setup of the ZED X One guides according to your hardware.

Install the ZED SDK #

The ZED SDK provides tools and libraries to process the camera stereoscopic data. In the case of the ZED X Ones, a tool, ZED Media Server, will be required to setup and use a pair of ZED X One cameras in order to use it as a standard ZED stereo camera input in the ZED SDK.

Download and install the ZED SDK on the NVIDIA Jetson.

Install the ZED X One driver #

The ZED X series requires installing a driver to operate correctly. It is responsible for configuring the GMSL2 device and is dependent on the hardware used, such as the Jetson carrier board, the deserializer card, and other configurations related to GMSL2 cameras.

The drivers can can be downloaded from the ZED X Driver download page.

Note: Please make sure to check the version of the driver available on the ZED X Driver download page. If the website features an updated version, we recommend installing the latest one.

Select the driver corresponding to your configuration and run:

sudo dpkg -i stereolabs-zedx_X.X.X-YYYYYYYY-L4TZZ.Z_arm64.deb

Where:

• X.X.X is the driver version
• YYYYYYY is the deserializer
• L4TZZ.Z is the Jetson Linux version

For example, for L4T 35.1 (JetPack 5.0) with the Stereolabs GMSL2 Capture Card, run:

sudo dpkg -i stereolabs-zedx_0.5.1-MAX96712-L4T35.1_arm64.deb

Note: You might need to install libqt5core5a if not already installed. Use the following command:

sudo apt install libqt5core5a

Now, reboot the NVIDIA® Jetson.

You can use the following command to check that the driver is correctly loaded:

sudo dmesg | grep zedx

Please note that GMSL2 cameras have lower flexibility compared to USB cameras. Any modification in the hardware configuration, such as plugging or unplugging a camera, or changing the camera order, requires either rebooting the Jetson, or restarting the installed daemon with the following command:

sudo systemctl restart zed_x_daemon

To troubleshoot issue, refer to this page.

sudo dpkg -r stereolabs-zedx_X.X.X-YYYYYYYY-L4TZZ.Z_arm64.deb

sudo dpkg -l | grep stereolabs-zedx

Configuring the Stereo system #

Hardware #

The ZED X One should be rigidly mounted parallel to each other at a fixed distance. It is critical that the cameras don’t move in rotation or translation relative to each other over time. Any movement will require a new calibration. The mechanical stability should be adapted depending on usage conditions, especially vibration or shocks that could impact cameras’ alignments and significantly decrease the accuracy of the system. More information on setting up the stereo rig can be found here.

Software #

To compute depth and 3D data using the ZED SDK, the ZED X One camera pairs need to be configured into one stereoscopic camera.

The first step is to configure a virtual Stereo camera from 2 ZED X One. Open the ZED Media Server tool to setup this system.

ZED_Media_Server

The GUI of ZED Media Server allows setting up which cameras should be the left and the right, using the cameras’ serial numbers. Please make sure that the system is correctly set up, especially that the left and right cameras are not swapped.

Set the desired resolution, then click the bottom SAVE button once the system is set up. This will store the configuration of the cameras serial numbers and their location in the virtual stereo system. The virtual camera serial number SN on the left will be the one used in the ZED SDK to reference this virtual stereo camera.

Click the Stream button to start streaming the images.

The GUI can be closed, a service (zed_media_server_cli) will automatically start in the background to stream data from the virtual stereo camera.

There is no restriction on the virtual camera setup, which allows for a ZED X One to be used in multiple virtual stereo cameras at once.

Please note that at this point the stereo camera images are available but require a calibration procedure to be usable. The calibration provides the ZED X Ones’ relative position to each other (known as extrinsic parameters) and can also refine or compute the ZED X Ones’ lens parameters such as focal, optical center, distortion (known as intrinsic parameters). These calibration parameters are used by the ZED SDK to compute depth information.

Calibration #

The calibration step is mandatory and critical to get accurate depth information. This process is used to find the camera’s intrinsic and extrinsic parameters. If the mechanical configuration is altered or modified, the calibration must be recomputed. In this case, only the extrinsic parameters (translations and rotations between the 2 cameras) are impacted, but the calibration process usually refines both to get an accurate estimation.

Contrary to other ZED cameras, the ZED X Ones are not factory calibrated, as the stereo system can be customized. This means that the user is responsible for this process, including calibration file management (see below).

Procedure #

The calibration procedure requires recording images of the system looking at a calibration target of known characteristics, such as a planar checkerboard of known NxM size and square size. The positions in the image are critical to accurately determine the calibration parameters.

To calibrate the ZED X One, download and compile and install the acquisition library for ZED X One, then compile and run the calibration sample program.

git clone https://github.com/stereolabs/zedx-one-capture.git
cd zedx-one-capture/lib ; mkdir build ; cd build ; cmake ..
sudo make -j6 install

Pattern #

We need a calibration pattern as a marker and as a world unit reference. You can refer to this page to create your own.

Calibration pattern good practices:

• it should be printed or stuck to a flat surface.
• it should be printed with accuracy, clear black and white, and no aliasing.
• its size should be significant, ideally A3 or more.
• it should have enough markers (>100).
• its marker size should be known and accurate.

Calibration #

Compile and Run the calibration sample available in the same git repository:

cd ../stereo_calibration
mkdir build ; cd build ; cmake ..
make -j6
./zedx_one_stereo_calibration

Make sure the printed chessboard parameters match the parameters in the code, it requires the chessboard width and height and square size in millimeters.

The calibration process uses images saved manually with the chessboard visible and at different positions each time in the image.

When calibrating, make sure the calibration pattern is seen in all the fields of view of the camera along the process.

Multiply various poses, close up, distant, tilted… The chessboard should be tilted along each axis at different distances, the GUI indicates which axis should be further excited.

At each pose, keep the camera still for a short period to avoid motion blur, the pattern should be fully visible in both the left and right images.

Once enough images are saved the calibration process will run automatically, and the calibration parameters will output as a configuration file readable by the SDK.

Calibration file #

The calibration process will generate a configuration file, this file should be named SN<virtual camera serial>.conf (like SN1025271012.conf) and placed in the settings folder of the ZED SDK, /usr/local/zed/settings/ on Linux.

Unlike the ZED or ZED X cameras, the file should be shared manually with every host computer that needs to open the camera system. Make sure that the calibration information is up to date in case the system has changed and was re-calibrated.

Using the ZED SDK #

The ZED X One stereo system can be opened with the ZED SDK by using the streaming input mode, on the same machine or remotely on a local network. The images are encoded and sent by the ZED Media Server tool (either by the GUI or the service).

To open a ZED X One stereo system, use the streaming address of the Jetson (127.0.0.1 for localhost) and the streaming port, typically 34000, in InitParameters.

Then, call open() to open the camera from the stream and grab() to grab a new frame and perform any type of processing.

// Set the input from stream
InitParameters init_parameters;
init_parameters.input.setFromStream("127.0.0.1", 34000); // Specify the IP and port of the sender
// Open the camera
ERROR_CODE err = zed.open(init_parameters);
if (err != ERROR_CODE::SUCCESS)
    exit(-1);

while (!exit_app) {
    if (zed.grab() == ERROR_CODE::SUCCESS) {
        // Any processing
    }
}
// Close the camera
zed.close();

# Set the input from stream
init = sl.InitParameters()
init.set_from_stream("127.0.0.1", 34000) # Specify the IP and port of the sender
# Open the camera
err = zed.open(init)
if err != sl.ERROR_CODE.SUCCESS:
	exit(1)

while not exit_app:
    if zed.grab() == sl.ERROR_CODE.SUCCESS:
        # Any processing

# Close the camera
zed.close()

// Set the input from stream
InitParameters initParameters = new InitParameters();
initParameters.inputType = INPUT_TYPE.STREAM; 
initParameters.ipStream = "127.0.0.1"; // Specify the IP of the sender
initParameters.portStream = "34000"; // Specify the port of the sender

RuntimeParameters runtimeParameters = new RuntimeParameters();
// Open the camera
ERROR_CODE err = zed.Open(ref initParameters);
if (err != ERROR_CODE.SUCCESS)
    Environment.Exit(-1);

while (!exit_app) {
    if (zed.Grab(ref runtimeParameters) == ERROR_CODE.SUCCESS) {
        // Any processing
    }
}
// Close the camera
zed.Close();

Code Example #

See the Camera Streaming samples for more information on streaming a video and using a stream as input.