Introducing the updated ZED GStreamer package
GStreamer is an open source, cross-platform multimedia development framework that makes it possible to write any type of streaming multimedia application.
GStreamer is an open source, cross-platform multimedia development framework that makes it possible to write any type of streaming multimedia application.
We are excited to announce the launch of the latest version of the GStreamer package that now supports the ZED SDK v4 along with the new ZED X and ZED X Mini GMSL2 cameras. Our GStreamer repository offers a range of GStreamer elements that make it convenient to incorporate a ZED camera into a GStreamer multimedia pipeline. Our plugins allow to capture RGB, Depth, Sensors, Object Detection, and Skeleton Tracking data, which can be manipulated and streamed, taking advantage of over 250 free GStreamer plugins that offer more than 1000 elements of manipulation.
The ZED GStreamer package
The ZED GStreamer package contains five plugins that offer five pipeline elements, along with a metadata library and a Linux RTSP Server application.
Plugins
You can use the five GStreamer elements as they are, or you can use them as a model to create new plugins for ZED data management.
zedsrc: source element that allows capturing camera color images, depth maps, and sensor data and sending them into a GStreamer pipeline. It enables the streaming of different types of data on a single buffer, including individual left/right/depth frames, a vertical composite frame containing both left and right frames, and a vertical composite frame containing a combination of the left frame and depth map. All streams include metadata containing sensor data, camera position, and information on object detection and skeleton tracking.zeddemux: de-muxer element that receives a compositezedsrcstream (color left + color rightorcolor left + depth map) with metadata, processes the eventual depth data, and splits the input into two separated new streams on two pads namedsrc_leftandsrc_aux. A third source pad is available for metadata to be processed by any other element.zeddatamux: muxer element that receives a video stream compatible with ZED caps, a ZED metadata stream generated by thezeddemuxand adds the metadata to the input video stream. This is useful if a GStreamer filter element that does not automatically propagate metadata removes them.zeddatacsvsink: example of a sink element that receives ZED metadata stream, extracts the Positional Tracking and the Sensors Data information, and saves them in a CSV file.
Metadata library
zedmeta: GStreamer library to define and handle the ZED metadata (Positional Tracking data, Sensors data, Detected Object data, Detected Skeletons data).
RTSP Server
gst-zed-rtsp-launch: a Linux-only application that instantiates an RTSP server from a text launch pipeline “gst-launch” like.
Installation
Installation and building guides for both Windows and Linux operating systems are provided with the README file available on the Github repository or available in the online documentation.
How to use
All the elements offered by the ZED plugins are readily available for use in any GStreamer pipeline. To begin, let’s start with the basics.
The easier way to use the ZED GStreamer plugins is by command line using the gst-launch application.
A very simple pipeline
gst-launch-1.0 zedsrc ! autovideoconvert ! queue ! fpsdisplaysin
This simple pipeline starts ZED acquisition using the default parameters of the zedsrc element and automatically converts (autovideoconvert) the frames to the correct format for the fpsdisplaysink element in order to be displayed on the screen with FPS information.
The queue element is used to decouple frames in a different queue thread, useful for data synchronization for more complex pipelines.
To explore all the possible options and parameters provided by the zedsrc element it is possible to use the gst-inspect command:
gst-inspect-1.0 zedsrc
You can use the same command to get information for all the other elements.
A more complex example: sending data over a local network
GStreamer is known for its ability to easily transmit media data over networks. It offers various protocols such as TCP, UDP, RTP, and encoding algorithms like H264, H265, VP8, VP9, Matroska, and more.
The following gst-launch pipeline acquires an RGB stream from a ZED camera, displays it on the local machine, and sends it over a network using the UDP protocol with H264 encoding:
gst-launch-1.0 zedsrc ! timeoverlay ! tee name=split has-chain=true ! \ queue ! autovideoconvert ! fpsdisplaysink \ split. ! queue max-size-time=0 max-size-bytes=0 max-size-buffers=0 ! autovideoconvert ! x264enc byte-stream=true tune=zerolatency speed-preset=ultrafast bitrate=3000 ! \ h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! \ udpsink clients=<ip_address>:5000 max-bitrate=3000000 sync=false async=false
- Line 1: the
zedsrcelement is configured to acquire the left camera RGB data with default resolution and framerate for the connected camera. The timestamp is printed on the image using thetimeoverlayelement and the pipeline is split into two branches usingtee. - Line 2: the first branch of the pipeline is used to display the frames on the local machine with FPS and timestamp information.
- Line 3: the second branch of the pipeline is used to encode the stream using the H264 encoder (
x264enc) configured to reduce the latency to the minimum possible. - Line 4: the H264 stream is parsed (
h264parse) and a RTP payload is created (rtph264pay). - Line 5: the RTP payload is sent using UDP (
udpsink) to the client at address<ip_address>that will be listening on port5000.
Note: the element queue max-size-time=0 max-size-bytes=0 max-size-buffers=0 is necessary to synchronize the two branches generated by the tee element, otherwise the stream gets frozen after a few milliseconds.
The receiver machine can acquire the stream using this simple pipeline:
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,clock-rate=90000,payload=96 ! \ queue ! rtph264depay ! h264parse ! avdec_h264 ! \ queue ! autovideoconvert ! fpsdisplaysink
- Line 1: a UDP source (
udpsrc) is configured to listen on port 5000 for an RTP payload of type 96. - Line 2: the payload is elaborated (
rtph264depay) to extract the H264 data to be parsed (h264parse) and then decoded (avdec_h264). - Line 3: the decoded stream is finally converted and displayed on the screen with FPS information.
Comparing visually the timestamp printed on each frame on the sender and on the receiver machines, it is possible to grossly evaluate the latency of the network.
Visualize object detection and skeleton data
The ZED GStreamer package not only provides data acquisition and stream manipulation elements but also a useful plugin to draw on each frame the results of the processing of the Object Detection and Skeleton Tracking modules of the ZED SDK.
An example pipeline is provided to enable the skeleton tracking module in the zedsrc element. The skeleton metadata is processed with the zedodoverlay element to render the results on each frame and the final result is displayed on the screen with FPS information.
gst-launch-1.0 zedsrc stream-type=0 bt-enabled=true bt-detection-model=2 ! queue ! zedodoverlay ! queue ! autovideoconvert ! fpsdisplaysink
The zedsrc element is configured to acquire the single left RGB image with default resolution and framerate for the connected camera. The skeleton tracking module is enabled to detect and track the skeleton of all the persons on the scene using the fast method (bt-detection-model=2).
The result of the detection is added to the stream buffer as zedmeta metadata. The zedodoverlay element retrieves the metadata from the buffer and draws the results on each frame, automatically performing the rescaling in case it is required (see the rescaling example).
The same pipeline can be used to perform object detection by simply enabling the correct module:
gst-launch-1.0 zedsrc stream-type=0 od-enabled=true od-detection-model=0 ! queue ! zedodoverlay ! queue ! autovideoconvert ! fpsdisplaysink
The detected objects, which are all humans in the case of the picture above, are enclosed by 2D bounding boxes. The ID and the class label are also shown on top of the bounding box.
It’s possible to enable both Object Detection and Skeleton Tracking at the same time to gather information on both objects and skeletons. These two features are not mutually exclusive.
Creating an RTSP Server
The46 Real Time Streaming Protocol (RTSP) is a network control protocol designed for use in entertainment and communications systems to control streaming media servers. The protocol is used for establishing and controlling media sessions between end points.
The ZED GStreamer package provides an easy way to create an RTSP Server starting from a “textual pipeline description”, in the same way as gst-launch works, using the new introduced CLI command gst-zed-rtsp-launch.
The usage of the command gst-zed-rtsp-launch is simple:
gst-zed-rtsp-launch [OPTION?] PIPELINE-DESCRIPTION
There are two available options:
-p,--port=PORT-> Port to listen on (default: 8554).-a,--address=HOST-> Host address (default: 127.0.0.1).
The following example creates an RTSP Server providing a zedsrc stream encoded with H264, using the default configuration parameters.
gst-zed-rtsp-server zedsrc ! videoconvert ! 'video/x-raw, format=(string)I420' ! x264enc ! rtph264pay pt=96 name=pay0
Note: It is mandatory to define at least one payload named pay0; it is possible to define multiple payloads if the pipeline is split into more branches, using an increasing index (i.e. pay1, pay2, …).
The RTSP stream can be acquired by any RTSP-capable receiver simply connecting to the URL: rtsp://<server_ip_address>:8554/zed-stream.
A GStreamer receiver can be instantiated using gst-launch:
gst-launch-1.0 rtspsrc location=rtsp://<server_ip_address>:8554/zed-stream latency=0 ! decodebin ! fpsdisplaysink
The decodebin element automatically detects the encoding of the incoming stream and construct the correct decoding pipeline.
In conclusion
In this blog post, we introduced the updated ZED GStreamer repository. We provided examples of how to use it for simple basic operations.
The possibilities of GStreamer are endless. Don’t hesitate to share your projects and applications on the Stereolabs community forum. We’re happy to provide support and advice if needed.
