GStreamer - ZED Camera Source

The ZED Camera Source, zedsrc GStreamer element, is the main plugin of the package, it allows injecting the ZED data in a GStreamer pipeline and getting the information provided by the ZED SDK.

Properties

Property
Meaning
 Flags Type Range Default
camera-resolution Camera Resolution readable, writable Enum “GstZedsrcResolution” (3):VGA (2):HD720 (1):HD1080 (0):HD2K 1,“HD1080”
camera-fps Camera frame rate readable, writable Enum “GstZedSrcFPS” (100):100FPS - only VGA resolution (60):60FPS - only VGA and HD720 resolutions (30):30FPS - only VGA, HD720 and HD1080 resolutions (15):15FPS - all resolutions 30,“30FPS”
stream-type Image stream type readable, writable Enum “GstZedSrcCoordSys” (0):Left_image [BGRA] - 8 bits- 4 channels Left image (1):Right_image [BGRA] - 8 bits- 4 channels Right image (2):Stereo_couple_up/down [BGRA] - 8 bits- 4 channels bit Left and Right (3):Depth_image [GRAY16_LE] - 16 bits depth (4):Left_and_Depth_up/down [BGRA] - 8 bits- 4 channels Left and Depth(image) 0,“Left_image [BGRA]”
sdk-verbose ZED SDK Verbose readable, writable Boolean true, false false
camera-image-flip Use the camera in forced flip/no flip or automatic mode readable, writable Enum “GstZedSrcFlip” (0):No_Flip (1):Flip (2):Auto 2,“Auto”
camera-id Select camera from cameraID readable, writable Integer 0 - 255 0
camera-sn Select camera from camera serial number readable, writable Integer64 0 - 9223372036854775807 0
svo-file-path Input from SVO file readable, writable String - ""
input-stream-ip Specify IP adress when using streaming input readable, writable String - ""
input-stream-port Specify port when using streaming input readable, writable Integer 1 - 65535 30000
depth-minimum-distance Minimum depth value readable, writable Float 100 - 3000 300
depth-maximum-distance Maximum depth value readable, writable Float 500 - 40000 20000
camera-disable-self-calib Disable the self calibration processing when the camera is opened readable, writable Boolean true, false false
depth-stabilization Enable depth stabilization readable, writable Boolean true, false true
confidence-threshold Specify the Depth Confidence Threshold readable, writable Integer 0 - 100 100
texture-confidence-threshold Specify the Texture Confidence Threshold readable, writable Integer 0 - 100 100
sensing-mode Specify the Depth Sensing Mode readable, writable Enum “GstZedsrcSensingMode” (0):STANDARD (1):FILL 0,“STANDARD”
measure3D-reference-frame Specify the 3D Reference Frame Enum “GstZedsrc3dMeasRefFrame” (0):WORLD (1):CAMERA 0,“WORLD”
enable-positional-tracking Enable positional tracking readable, writable Boolean true, false true
set-as-static Set to TRUE if the camera is static readable, writable Boolean true, false false
area-file-path Area localization file that describes the surroundings, saved from a previous tracking session readable, writable String - ""
enable-area-memory This mode enables the camera to remember its surroundings. This helps correct positional tracking drift, and can be helpful for positioning different cameras relative to one other in space. readable, writable Boolean true, false true
enable-imu-fusion This setting allows you to enable or disable IMU fusion. When set to false, only the optical odometry will be used. readable, writable Boolean true, false true
enable-pose-smoothing This mode enables smooth pose correction for small drift correction. readable, writable Boolean true, false true
set-floor-as-origin This mode initializes the tracking to be aligned with the floor plane to better position the camera in space. readable, writable Boolean true, false false
initial-world-transform-x X position of the camera in the world frame when the camera is started. readable, writable Float -3.402823e+38 - +3.402823e+38 0.0
initial-world-transform-y Y position of the camera in the world frame when the camera is started. readable, writable Float -3.402823e+38 - +3.402823e+38 0.0
initial-world-transform-z Z position of the camera in the world frame when the camera is started. readable, writable Float -3.402823e+38 - +3.402823e+38 0.0
initial-world-transform-roll Roll orientation of the camera in the world frame when the camera is started. readable, writable Float 0.0 - 360.0 0.0
initial-world-transform-pitch Pitch orientation of the camera in the world frame when the camera is started. readable, writable Float 0.0 - 360.0 0.0
initial-world-transform-yaw Yaw orientation of the camera in the world frame when the camera is started. readable, writable Float 0.0 - 360.0 0.0
coordinate-system 3D Coordinate System readable, writable Enum “GstZedsrcStreamType” (0):Image (1):Left_handed,Y_up (2):Right_handed,Y_up (3):Right_handed,Z_up (4):Left_handed,Z_up (5):Right_handed,Z_up,X_fwd 0,“Image”
od-enabled Set to TRUE to enable Object Detection readable, writable Boolean true, false false
od-image-sync Set to TRUE to enable Object Detection frame synchronization readable, writable Boolean true, false true
od-tracking Set to TRUE to enable tracking for the detected objects readable, writable Boolean true, false true
od-detection-model Object Detection Model readable, writable Enum “GstZedSrcOdModel” (0):Object_Detection_Multi_class (1):Object_Detection_Multi_class_ACCURATE (2):Skeleton_tracking_FAST (3):Skeleton_tracking_ACCURATE (4):Object_Detection_Multi_class_MEDIUM (5):Skeleton_tracking_MEDIUM 0,“Object_Detection_Multi_class”
od-confidence Minimum Detection Confidence readable, writable Float 0 - 100 50
od-max-range Maximum Detection Range readable, writable Float 0 - 20000 20000
od-body-fitting Set to TRUE to enable body fitting for skeleton tracking readable, writable Boolean true, false true
brightness Image brightness readable, writable Integer64 0 - 8 4
contrast Image contrast readable, writable Integer64 0 - 8 4
hue Image hue readable, writable Int 0 - 11 0
saturation Image saturation readable, writable Integer64 0 - 8 4
sharpness Image sharpness readable, writable Integer64 0 - 8 4
gamma Image gamma readable, writable Integer64 1 - 9 8
gain Camera gain readable, writable Integer64 0 - 100 60
exposure Camera exposure readable, writable Integer64 0 - 100 80
aec-agc Enable camera automatic gain and exposure readable, writable Boolean true, false true
aec-agc-roi-x Auto gain/exposure ROI top left ‘X’ coordinate (-1 to not set ROI) readable, writable Integer64 -1 - 2208 -1
aec-agc-roi-y Auto gain/exposure ROI top left ‘Y’ coordinate (-1 to not set ROI) readable, writable Integer64 -1 - 1242 -1
aec-agc-roi-w Auto gain/exposure ROI width (-1 to not set ROI) readable, writable Integer64 -1 - 2208 -1
aec-agc-roi-h Auto gain/exposure ROI height (-1 to not set ROI) readable, writable Integer64 -1 - 1242 -1
aec-agc-roi-side Auto gain/exposure side (-1 to not set ROI) readable, writable Enum (0): Left side only - LEFT (1): Right side only - RIGHT (2): Left and Right side - BOTH 2,“BOTH”
whitebalance-temperature Image white balance temperature readable, writable Integer64 2800 - 6500 4600
whitebalance-auto Enable camera automatic white balance temperature readable, writable Boolean true, false true
led-status Camera LED on/off readable, writable Boolean true, false true

Example pipelines

Display left image on screen

This simple pipeline starts the ZED grabbing using the default parameters of the zedsrc element and automatically converts (autovideoconvert) the frames to the correct format for the fpsdisplaysink element in order to be displayed on screen with FPS information.

gst-launch-1.0 zedsrc ! autovideoconvert ! queue ! fpsdisplaysink

The queue element is used to decouple frames in a different queue thread, useful for data synchronization for more complex pipelines.

Display color stereo couple with VGA resolution at 100 FPS

This pipeline is the same as the one of the previous example, but the properties of zedsrc are changed to set VGA resolution and 100 FPS retrieving the synchronized stereo image.

gst-launch-1.0 zedsrc camera-resolution=3 camera-fps=100 stream-type=2 ! autovideoconvert ! queue ! fpsdisplaysink

Display depth map with HD720 resolution at 60 FPS

Again the same pipeline, but the parameters are changed to retrieve only the not normalized depth map with HD720 resolution at 60 FPS.

gst-launch-1.0 zedsrc camera-resolution=2 camera-fps=60 stream-type=3 ! autovideoconvert ! queue ! fpsdisplaysink

H264 streaming over UDP on local network

GStreamer is famous for its capability of providing an easy way of transmitting media data over networks. TCP, UDP, RTP and many other protocols are available, together with many encoding algorithms like H264, H265, VP8, VP9, matroska, ecc.

The following gst-launch pipeline acquires an RGB stream from a ZED camera, displays it on the local machine and send it over network using the UDP protocol with H264 encoding:

gst-launch-1.0 zedsrc camera-resolution=2 camera-fps=30 ! timeoverlay ! tee name=split has-chain=true ! \
 queue ! autovideoconvert ! fpsdisplaysink \
 split. ! queue max-size-time=0 max-size-bytes=0 max-size-buffers=0 ! autovideoconvert ! \
 x264enc byte-stream=true tune=zerolatency speed-preset=ultrafast bitrate=3000 ! \
 h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! \
 udpsink clients=192.168.1.169:5000 max-bitrate=3000000 sync=false async=false

  • Line 1: the zedsrc element is configured to acquire the left camera RGB data with a resolution of 1280×720 (HD720) at 30 FPS. The timestamp is printed on the image using the timeoverlay element and the pipeline is splitted into two branches using tee.
  • Line 2: the first branch of the pipeline is used to display the frames on the local machine with FPS and timestamp information.
  • Line 3: the second branch of the pipeline is used to encode the stream using the H264 encoder (x264enc) configured to reduce the latency to the minimum possible.
  • Line 4: the H264 stream is parsed (h264parse) and a RTP payload is created (rtph264pay).
  • Line 5: the RTP payload is sent using UDP (udpsink) to the client at address 192.168.1.169 that will be listening on port 5000.

Note: the element queue max-size-time=0 max-size-bytes=0 max-size-buffers=0 is necessary to synchronize the two branches generated by the tee element, otherwise the stream gets frozen after a few milliseconds.

The receiver machine can acquire the stream using this simple pipeline:

gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,clock-rate=90000,payload=96 ! \
 queue ! rtph264depay ! h264parse ! avdec_h264 ! \
 queue ! autovideoconvert ! fpsdisplaysink

  • Line 1: an UDP source (udpsrc) is configured to listen on port 5000 for a RTP payload of type 96.
  • Line 2: the payload is elaborated (rtph264depay) to extract the H264 data to be parsed (h264parse) and then decoded (avdec_h264).
  • Line 3: the decoded stream is finally converted and displayed on screen with FPS information.

Comparing visually the timestamp printed on each frame on the sender and on the receiver machines, it is possible to grossly evaluate the latency of the network.