ZED SDK 4.0

What’s New

We are excited to announce the release of ZED SDK 4.0, which introduces a range of new features and enhancements to our ZED cameras. Our latest update supports the ZED X and ZED X Mini cameras, designed specifically for autonomous mobile robots in indoor and outdoor environments. We are also introducing an improved NEURAL depth mode, which offers even more accurate depth maps in challenging situations such as low-light environments and textureless surfaces.

We are proud to introduce the new multi-camera Fusion API, which makes it easier than ever to fuse data coming from multiple cameras. This module handles time synchronization and geometric calibration issues, along with 360° data fusion with noisy data coming from multiple cameras and sensor sources. We believe that these updates will unlock even more potential for our users to create innovative applications that push the boundaries of what is possible with depth-sensing technology.

For those upgrading from SDK version 3.X to 4.0, we highly recommend checking out our migration guide to ensure a smooth transition.

Major 4.0 API Changes

Video

• Added new camera models MODEL::ZED_X and MODEL::ZED_XM.
• Added the new INPUT_TYPE::GMSL input type for GMSL cameras (support for GMSL cameras is only available on hosts with Nvidia Jetson SoC).
• Added INPUT_TYPE input_type field to sl::DeviceProperties struct.
• Added Camera reboot non-blocking behavior. This is enabled using the new InitParameters::async_grab_camera_recovery parameter. When the asynchronous automatic camera recovery is enabled and there’s an issue with the communication with the camera, the grab() will exit after a short period of time, and return the new ERROR_CODE::CAMERA_REBOOTING error code. The recovery will run in the background until proper communication is restored. The default and previous ZED SDK behavior is synchronous, the grab() function is blocking and will return only once the camera communication is restored or the timeout has been reached.
• setFromCameraID and setFromSerialNumber methods from sl::InputType now take an optional BUS_TYPE to choose between USB or GMSL cameras. When unspecified, the method searches for available USB cameras first, then searches for GMSL.
• Added new methods getInputType(), getConfiguration(), and isInit() for sl::InputType.
• Added a scale utility method to sl::CameraParameters to easily convert camera intrinsic parameters for a given resolution.
• Added RESOLUTION::HD1200 (1920x1200) and RESOLUTION::SVGA (960x600) resolutions for ZED X and ZED X Mini cameras.
• Added RESOLUTION::AUTO resolution which sets RESOLUTION::HD720 resolution for USB cameras, and RESOLUTION::HD1200 resolution for GMSL cameras.
• Added new parameters in VIDEO_SETTINGS for GMSL cameras only:
  • EXPOSURE_TIME: Image sensor exposure time in ms
  • ANALOG_GAIN: Analog sensor gain in dB
  • DIGITAL_GAIN: Digital ISP gain in dB
  • AUTO_EXPOSURE_TIME_RANGE: Defines the range of the exposure time in automatic control
  • AUTO_ANALOG_GAIN_RANGE: Defines the range of sensor gain in automatic control. Min/Max range between [1000 - 16000] mdB.
  • AUTO_DIGITAL_GAIN_RANGE: Defines the range of digital ISP gain in automatic control.
  • EXPOSURE_COMPENSATION: Exposure target compensation made after AE. Reduces the overall illumination by a factor of F-stops. values range is [0 - 100] (mapped between [-2.0, 2.0]).
  • DENOISING: Defines the level of denoising applied on both left and right images. values range is [0 - 100].
• Camera::getCameraSettings now returns an ERROR_CODE and uses a reference to retrieve the value.
• Camera::setCameraSettings now returns an ERROR_CODE instead of void.

Depth

• SENSING_MODE has been removed from the API. To get the same behavior, use the new sl::RuntimeParameters.enable_fill_mode parameter.
• Improved Fused Point Cloud extraction time by 2.5x.
• Improved Positional Tracking loop closure time.
• Improved Windows AI model download error handling: when there’s no internet access or if the server can’t be reached the exact error is now displayed. It will also auto-retry an alternative link if the primary server can’t be reached.

Multi-Camera Fusion API

• Added sl::Fusion API in a new header sl/Fusion.hpp. In 4.0 Early Access, multi-camera fusion supports multi-camera capture, calibration, and body tracking fusion. All SDK capabilities will be added to the Fusion API in the final 4.0 release.
• The Fusion API introduces publishers and subscribers to exchange data between source cameras and the fusion module.
• A publisher will perform its computations and publish its data using the startPublishing() method.
• A subscriber will use the subscribe() method from the sl::Fusion class to connect to all of the publishers and start retrieving data. The sl::Fusion.process() method takes care of all the heavy lifting of synchronizing and fusing the data, and produces the enhanced body tracking information in real-time.
• Local workflow: sl::Fusion is available for a setup with multiple cameras on a single host, using intra-process communication. The publishers and the single subscriber run on the same machine in this configuration.
• Distributed workflow: sl::Fusion is available for a setup with multiple cameras on multiple hosts, communicating on the local network, enabled by ZED Hub. Each publisher and the single subscriber run on separate machines such as ZED Boxes, in order to distribute computations with devices on the edge.

Geo-tracking API

• The Geo-tracking API is a new module that brings global-scale location tracking to the ZED SDK. It improves the ZED camera’s positional tracking capabilities by incorporating additional GNSS data to achieve global positioning. This is done by utilizing the new sl::Fusion.ingestGNSSdata method, which integrates the GNSS data with the existing tracking data to create a fused position. The resulting position can be accessed both locally and globally using the getPosition and getGeoPose methods, respectively. With this capability, users can achieve highly accurate positioning information that can be used in a variety of applications.
• Added a GeoPose structure containing the tracked position in global world reference.
• Added a GNSSData structure containing information about GNSS input data.
• Added structures ECEF, LatLng, and UTM for different GPS/GNSS coordinate systems.
• Added getGeoPose and getPosition methods in sl::Fusion that return the fused position in the local and global world frames.
• Added Geo2Camera and Camera2Geo methods in sl::Fusion to convert from a position in the global world reference (world map) frame to a position in the local world frame, and vice-versa.

Body Tracking

• Body Tracking is now a standalone module, with its own specific parameters and functions.
• The new BODY_TRACKING_MODEL enum contains the following models:
  • HUMAN_BODY_FAST: Keypoints based, specific to human skeleton, real-time performance even on Jetson or low-end GPU cards.
  • HUMAN_BODY_MEDIUM: Keypoints based, specific to human skeletons, compromise between accuracy and speed.
  • HUMAN_BODY_ACCURATE: Keypoints based, specific to human skeletons, state-of-the-art accuracy, requires powerful GPU.
• The model now depends both on the accuracy wanted and on the BODY_FORMAT (see below).
• Changed BODY_FORMAT namings from POSE_XX to BODY_XX. The POSE_18 and POSE_34 formats have been renamed to BODY_18 and BODY_34 respectively.
• Compared to ZED SDK 3.8, the HUMAN_BODY_FAST and HUMAN_BODY_ACCURATE models when using BODY_18 and BODY_34 are slower but more precise. Up to 50% more precise for the fast model with a runtime difference of around 2 ms on Jetson Orin.
• Added a new human body format in BODY_FORMAT:
  • BODY_38: Adds additional key points to BODY_34, specifically on the feet and hands. The body fitting now provides accurate orientations for the feet and hands.
• BODY_18 and BODY_34: The previous body models are still available
• Added the possibility to use multiple body tracking models at once using the instance_id parameter in sl::BodyTrackingParameters (and in related methods such as Camera::getBodyTrackingParameters, Camera::disableBodyTracking, …).
• Added a BODY_KEYPOINTS_SELECTION enum to filter key points for specific use cases. Currently available options include BODY_KEYPOINTS_SELECTION::FULL and BODY_KEYPOINTS_SELECTION::UPPER_BODY.
• Analogous to object detection, body-tracking objects now have their own methods and parameters exposed separately:
  • Added a new sl::BodyData class which contains detection and tracking information on detected human bodies, such as keypoint, joint, and skeleton root information.
  • Added a new Camera::retrieveBodies method to retrieve human body detection data after a Camera::grab call, similar to Camera::retrieveObjects.
  • Added a new sl::Bodies class which is returned from Camera::retrieveBodies, similar to sl::Objects.
  • Added a new sl::BodyTrackingParameters and sl::BodyTrackingRuntimeParameters classes which define all of the parameters related to body tracking.
• Renamed BODY_PARTS to BODY_18_PARTS.
• Renamed BODY_PARTS_POSE_34 to BODY_34_PARTS.
• Added BODY_38_PARTS which lists all key points for the new body format.
• Renamed BODY_BONES to BODY_18_BONES.
• Renamed BODY_BONES_POSE_34 to BODY_34_BONES.
• Added BODY_38_BONES bones containers for new body format.

Object Detection

• Object detection models have been separated into object detection models and human body detection models, in two separate classes: OBJECT_DETECTION_MODEL and BODY_TRACKING_MODEL respectively.
• The new OBJECT_DETECTION_MODEL enum contains the following models:
  • MULTI_CLASS_BOX_FAST : Any objects, bounding box based.
  • MULTI_CLASS_BOX_MEDIUM : Any objects, bounding box based, a compromise between accuracy and speed.
  • MULTI_CLASS_BOX_ACCURATE : Any objects, bounding box based, more accurate but slower than the base model.
  • PERSON_HEAD_BOX_FAST : Bounding Box detector specialized in the person’s head, particularly well suited for crowded environments, the person localization is also improved.
  • PERSON_HEAD_BOX_ACCURATE : Bounding Box detector specialized in the person’s head, particularly well suited for crowded environments, the person localization is also improved, more accurate but slower than the base model.
  • CUSTOM_BOX_OBJECTS : For external inference, by using your own custom model and/or frameworks. This mode disables the internal inference engine, the 2D bounding box detection must be provided.
• Added the possibility to use multiple object detection models instances at once using the instance_id parameter in sl::ObjectDetectionParameters (and in related methods such as Camera::getObjectDetectionParameters, Camera::disableObjectDetection , …).
• Renamed MULTI_CLASS_BOX to MULTI_CLASS_BOX_FAST in OBJECT_DETECTION_MODEL .
• Renamed PERSON_HEAD_BOX to PERSON_HEAD_BOX_FAST in OBJECT_DETECTION_MODEL .
• Renamed enable_mask_output to enable_segmentation in sl::ObjectDetectionParameters .
• Removed keypoint_2d, keypoint, keypoint_confidence, local_position_per_joint, local_orientation_per_joint, and global_root_orientation from sl::ObjectData . These attributes have been moved to the sl::BodyData class for Body Tracking.

Known Issues

• Maxwell and Kepler GPUs (compute capabilities 35, 50, and 52) are not supported using CUDA 11.x, only with CUDA 10.2

4.0.8

SDK

• Improved Visual-Inertial Positional Tracking for ZED X.
• The Magnetometer heading rotation now compensates for the rotation.
• Resolved ZED X service access issues in Docker.
• Improved GeoTracking initialization process.

Bug Fixes

• Improved the reliability of concurrent camera openings on Linux.
• Fixed Camera2Geo returning a wrong geoposition when position is not in sl::COORDINATE_SYSTEM::IMAGE.
• Fixed a random memory leak in GeoTracking during initialization.
• Improved the reliability of the sl::Camera::reboot function.
• Fixed sl::Camera::reboot function, which was consistently returning CAMERA_NOT_DETECTED on Windows.
• Fixed a tracking issue with custom object detection that resulted in poor results with multiple small objects.
• Fixed an issue with sl::Camera::resetPositionalTracking to keep the previous state of the parameter set_gravity_as_origin.
• Improved the camera opening time that could increase a lot when using multiple cameras on Windows.

4.0.7

SDK

• Added an automatic SVO repair mechanism when opening a corrupted SVO file. The repair is done in place if possible to avoid duplication.
• Added automatic ROI exclusion computation of static objects. It requires motion to determine which part of the image is moving with the camera. This is typically used when the ZED is on a moving vehicle with a visible part in the image. A new sl::Camera::startRegionOfInterestAutoDetection function is used to initiate the detection. With enough motion and frames grabbed, it will be automatically applied by default and also available with sl::Camera::getRegionOfInterest. sl::RegionOfInterestParameters have been added to fine-tune the detection.
• Improved the newly released Positional Tracking Mode QUALITY to make it work faster and with all depth modes.
• Improved depth confidence to more accurately estimate low-quality depth pixels, such as flying pixels.
• Improved USB reliability on Linux by increasing the camera V4L2 priority. This should help avoid dropping frames, especially in case of heavy CPU loads.
• Improved the runtime of the Spatial Mapping module, especially on Jetson. On Orin AGX, the process from data integration to mesh retrieval can be up to 10 times faster compared to previous versions.
• Timestamps and dates are now logged in UTC format.
• Added sl::Camera::isCameraSettingSupported method for camera video control support.
• Added a new optional corrupted frame detector. This experimental feature can be enabled with sl::InitParameters::enable_image_validity_check. It is performed in the sl::Camera::grab() method and requires some computations. If an issue is found, the method will output a warning as the new sl::ERROR_CODE::CORRUPTED_FRAME. It’s currently able to catch USB corrupted frames such as green or purple-tinted frames or buffer corruption. It could also interpret widely different left and right images as invalid.

Fusion

• Added a sl::Fusion::stopPublishing method to the Fusion API.
• Reduced camera grab time when publishing data to Fusion.
• Added synchronized images and depth retrieval from within the Fusion API.

Bug Fixes

• Fixed incorrect GNSS calibration when the main camera is not at the origin.
• Fixed the fusion blocking when the publisher closes and reopens.
• Fixed a crash in the SDK logger in verbose mode.
• Fixed Positional Tracking that could miss movements in some situations.
• Fixed sl::Fusion::getPosition returning incorrect output when multiple cameras are set up.

Support

• Added L4T 35.4 compatibility.

4.0.6

SDK

• Improved GNSS-VIO Calibration and fusion.
  • Added several functions to make it easier to use: sl::Fusion::getCurrentGNSSCalibrationSTD / sl::Fusion::getGeoTrackingCalibration.
  • Added new options to control its behavior: target_translation_uncertainty / target_yaw_uncertainty / enable_rolling_calibration. Added new GNSS_CALIBRATION_STATE to get more details about its current state.
  • Added timestamp into GeoPose.
• Added function sl::Fusion::getCurrentTimeStamp to retrieve the timestamp of the current processed data.
• Added Spatial Mapping module into Fusion API. You can now map your environment with multiple ZED cameras at the same time.
• Improved Spatial Mapping accuracy, better reconstruction of shapes, improved detail when getting closer.
• Added new attribute sl::CameraParameters::focal_length_metric that stores the real focal length in millimeters.
• Added sl::PlaneDetectionParameters to better control the plane at hit function.
• Improved GMSL image capture pipeline, requires fewer resources and improves the latency.
• Improved GMSL SVGA resolution stability.
• Added support for specific carrier board (that are not using muxer before deserializer).
• Switched USB to max priority on Linux (previously “2” = INTERACTIVE, now “3” = RECORD).
• Improved USB stability on Linux.
• Added GMSL autorecovery.

Bug Fixes

• Fixed random issue when opening simultaneously multiple cameras.
• Fixed random camera motion sensor not detected during camera opening.
• Fixed timeout issue and automatic recovery when using 2 ZED X simultaneously.
• Fixed SVO decoding random issue on H26X lossless SVO on Jetsons.
• Fixed SVO recording missing IMU data for ZED X.
• Fixed missing parameters in sl::Camera::getInitParameters in SVO playback mode.

4.0.5

SDK

• Introduced low ZED X camera framerate, adding 15 FPS for all available resolutions. Requires Jetson driver update (v0.5.4)
• Added a new sl::InitParameters::grab_compute_capping_fps parameter to set an upper computation frequency limit to the grab function. This can be useful to get a known constant fixed rate or limit the computation load while keeping a short exposure time by setting a high camera capture framerate for instance.

Bug Fixes

• Resolved a bug that caused the GMSL IMU orientation to become invalid (not a number or not a valid quaternion) in some cases. This could also lead to SVO file reading issue since the frames were considered corrupted and skipped.

4.0.4

SDK

• Introduced a new sl::POSITIONAL_TRACKING_MODE, set with sl::PositionalTrackingParameters::mode. By default the STANDARD mode is enabled and identical to the previous implementation. The new QUALITY mode enables higher accuracy, especially in challenging cases such as low-feature environments or repetitive textures. Please note that it requires more computation and is for now only available with ULTRA depth mode.
• Improved Custom Object Detection module with better error logging and smarter diagnostic. Added security to avoid getting stuck when no depth is computed, invalid box input, invalid multi-instance id, and added an internal timeout when no data are available.
• Added color output to Spatial Mapping Mesh
• Added stability counter parameter in Spatial Mapping module. This defines the number of times each point has to be seen to be integrated into the spatial model. Decreasing the value will increase reactivity and integration speed at the cost of accuracy and noise increase.
• Improved sl::Camera::setRegionOfInterest function, it now accepts multiple channel images and provides a warning when the format is invalid instead of undefined behavior. Please note that it’s internally converted to grayscale images and only values equal to 0 are ignored.
• Added new parameter sl::BodyTrackingRuntimeParameters::skeleton_smoothing to configure the smoothing of the fitted skeleton, which ranges from 0 (low smoothing) to 1 (high smoothing).

Bug Fixes

• Fixed an issue that could occur with NEURAL depth mode where black vertical lines would corrupt the depth map.
• Fixed regression in object detection module with MULTI_CLASS_BOX* models on Jetson L4T 32.7, introduced in 4.0.3.

4.0.3

SDK

• Added GMSL support for NVIDIA Jetson L4T 35.3 (JetPack 5.1.1).
• Fixed new ZED Mini rectification that could lead to an image artifact.
• Improved body fitting for sl::BODY_FORMAT::BODY_38, it is now smoother and prevents the model from getting stuck in an unusual position.
• Reduced body jittering when standing still.
• Improved Object Detection accuracy.
• Improved AI overall loading time.

4.0.2

SDK

• Added new sl::FUSION_ERROR_CODE to provide more detailed feedback while using the Geo-tracking module.
• Updated default streaming bitrate to keep a constant quality regardless of the selected resolution/FPS/encoder.

Bug Fixes

• Fixed Geo-tracking module initialization.
• Fixed an issue leading to inconsistent hip positions with the body tracking Fusion module.
• Fixed body tracking ACTION_STATE parameter when body tracking is not activated.

Support

• Added CUDA 12.1 support for Windows.

4.0.1

Bug Fixes

• Fixed body tracking fusion with sl::BODY_FORMAT::BODY_34.
• Fixed inconsistent behavior of sl::BodyTrackingFusionRuntimeParameters::enable_body_fitting.
• Fixed sl::BodyTrackingFusionRuntimeParameters::skeleton_smoothing to be applied to sl::BODY_FORMAT::BODY_38.
• sl::Camera::getCameraSettings and sl::Camera::setCameraSettings now return sl::ERROR_CODE for non available settings (particularly with ZED X cameras).
• Fixed logger file creation without write permission.

Support

• Jetson running on JetPack 5.1 (L4T 35.2) now supports ZED X cameras.
• Added ZED SDK support for JetPack 5.1.1 (L4T 35.3), ZED X support coming soon.
• Added CUDA 12.0 partial support, CUDA 12.1 general support coming soon.

4.0.0 (Early Access)

ZED X Drivers

Added support for the new ZED X and ZED X Mini cameras.

Introduced new VIDEO_SETTINGS, RESOLUTION, and sl::InputType parameters to enable users to use the same code for both GMSL or USB 3.0 cameras without any added complexity.

Multi-Camera Fusion

Introduced new Multi-Camera Fusion API. The new module allows seamless synchronization and integration of data from multiple cameras in real time, providing more accurate and reliable information than a single camera setup. Additionally, the Fusion module offers redundancy in case of camera failure or occlusions, making it a reliable solution for critical applications.

The new API can be found in the header sl/Fusion.hpp as sl::Fusion API. In 4.0 Early Access, multi-camera fusion supports multi-camera capture, calibration, and body tracking fusion. All SDK capabilities will be added to the Fusion API in the final 4.0 release.