SDK

Platforms

• Introduced Ubuntu 20.04 and CUDA 11 support.
• Added support for NVIDIA JetPack 4.4 official release.

Improvements

• Improved object detection accuracy and stability, increasing AP by +15%. Object boxes are now more stable both in detection and tracking modes.
• Improved skeleton tracking: 3D localization of skeletons has been improved, along with their bounding boxes and head position. Skeletons with unrealistic sizes have also been removed. Body tracking is still in Beta, and further improvements will be released in future updates.
• Improved playback of SVO recorded in Flip mode. They will be automatically flipped when necessary.
• Improved AI model optimization process on Jetson platforms.

Bug Fixes

• Fixed a camera freeze issue on Jetson caused by the internal USB hub of the board. Now the camera will reconnect automatically if the hub disconnects the device.
• Fixed a bug in confidence of detected objects. When detection_confidence_threshold was changed, an object that was previously tracked could still be output even if the confidence was below the new threshold value. They are now removed.
• Fixed a H264/5 decoding issue when playing back SVO on Jetson platforms. Setting a frame position could previously lead to an SVO freeze.
• Fixed a positional tracking bug when using Local Streaming as input. An invalid tracking state was generated that would lead to wrong camera pose in Streaming mode.
• Fixed setSVOPosition() on Jetson platforms that was generating a wrong image sequence when grab() was called.
• Fixed a bug on sl::Mat initialisation by reference if that reference was empty. Now the sl::Mat created will be seen as uninitialized.
• Fixed an incorrect CMake output into stderr for a status message, reported by @spiderkeys.
• Fixed the performance optimizer script for Jetson NX where an incorrect power profile was selected.

Samples

• Added new sample "Point Cloud Mapping" to show how to map an area in point cloud format with the spatial mapping module.
• Fixed an OpenGL scaling issue in Body Tracking / Object detection samples.
• Fixed various bugs in samples.

New Features

Body Tracking (beta release)

• Added a new AI-powered detection model that detects and tracks body skeletons in 2D and 3D. Skeleton tracking can be used on both desktop and embedded platforms such as NVIDIA Jetson.
• Body tracking is part of the Object Detection module and can be called using DETECTION_MODEL::HUMAN_BODY_FAST or HUMAN_BODY_ACCURATE for a more accurate skeleton tracking at a lower FPS.
• Added new objects in the ObjectData class that contain skeleton and head data:
  • keypoint for 3D human keypoints/joints
  • head_bounding_box for head 3D bounding box
  • keypoint_2d for 2D human keypoints/joints
  • head_bounding_box_2d for head 2D bounding box
  • head_position for head 3D location
• Added 3D uncertainty (covariance) of a body root position in ObjectData::position_covariance[6].

Object Detection

• Improved Object Detection FPS on NVIDIA Jetson (requires JetPack 4.4). Detection and tracking is now 2x times faster on Jetson Nano compared to 3.1.
• Added 3D uncertainty (covariance) of an object position in ObjectData::position_covariance[6].

Video Streaming & Recording

• Added direct stream-to-record feature to allow recording from a streaming input without re-encoding the video. This allows to dramatically reduce hardware resources when ingesting and recording multiple streams on a single PC. Use RecordingParameters::transcode_streaming_input = false.
• Added framerate parameter for video recording and streaming. It is now possible to capture at 60FPS for example and record or stream at 15FPS. See RecordingParameters::target_framerate and StreamingParameters::target_framerate.
• Added bitrate parameter for SVO recording with specific bitrate values (for H264/H265 only). See RecordingParameters::bitrate.
• Added new Auto Flip mode for ZED 2 and ZED Mini (using IMU). Using InitParameter::camera_image_flip = FLIP_MODE::AUTO will automatically set the correct flip mode for the camera, so that you don't need to set it manually either in Live or SVO mode.
  This parameter will also works for SVO recorded with 3.X SDK version, since the flip mode was already recorded in the metadata. See InitParameter::camera_image_flip = FLIP_MODE::AUTO.

SDK

General Improvements

• Improved floor plane detection for ZED Mini and ZED 2 (using IMU).
• Improved barometer readings with sensor fusion to reduce drift during camera movements. It requires a ZED 2 sensor firmware update to v777.
• Updated toVerbose() function. Added CUDA error code/string in verbose message when ERROR_CODE::CUDA_ERROR is returned.

Bug Fixes

• Fixed IMU data that was not flipped correctly when flip was enabled with camera_image_flip.
• Fixed bug in IMU orientation covariance, leading to NaN occurences.
• Fixed grab() error code that could generate an ERROR_CODE::CAMERA_NOT_INITIALIZED at the end of an SVO file read in real time mode. It now generates a correct ERROR_CODE::END_OF_SVO_FILE.

Plugins & Interfaces

• Introduced a new C# wrapper for creating C# applications with the ZED SDK (Windows only). The C# API is now available on GitHub.
• Introduced a new open source capture library for the ZED camera family that lets you use the cameras on any hardware platform. It provides low level access to the camera video and sensors, as well as camera controls such as exposure/gain, which were not accessible with standard capture libraries (like OpenCV).
  The project is available on GitHub as zed-open-capture.

Platforms

• Introduced Jetson Xavier NX and JetPack 4.4 support.

Known Issues

• Static libraries are not available on JetPack 4.4 due to a linking issue with TensorRT static library. This should be fixed in the next patch.
• Updating ZED 2 sensor firmware on Windows can sometimes be unreliable. While we are working on a fix, we recommend updating your camera firmware on Linux.