ZED Topic Benchmark Tool
The ZED Benchmark tool measures the real performance of the topics published by the ZED ROS 2 nodes: how many messages per second actually reach a subscriber and how much bandwidth they consume. It is the recommended way to verify that a machine, a network, and a DDS configuration are able to deliver the camera data at the expected rate, and to quantify the benefit of Composition and IPC.
The tool is distributed in the tools/benchmark folder of the zed-ros2-examples repository.
For each received message the following information is available:
- Topic type
- Instantaneous frequency
- Windowed average frequency
- Message size
- Instantaneous topic bandwidth
- Windowed average topic bandwidth
The values are printed on the console in real time and published on a dedicated topic, so they can be plotted with tools like PlotJuggler or rqt_plot. When the test ends, a final report with the aggregated statistics of the whole session is printed and, optionally, saved to a file.
The benchmark node subscribes to the topic using a generic subscription, so it works with any ROS 2 topic, not only with the topics published by the ZED nodes.
The tool is available for ROS 2 Humble Hawksbill, Jazzy Jalisco, Lyrical Luth, and newer distributions. It is not available for ROS 2 Foxy and older releases, because they lack the generic subscription API required to subscribe to topics of an arbitrary type. On those distributions the three packages are still registered in the ament index, so the workspace build succeeds, but nothing is produced: no message, no component library, and no zed_topic_benchmark executable. A warning is printed by CMake during the build.
The packages
Installation
The benchmark packages are built together with the other packages of the zed-ros2-examples repository:
If you only need the benchmark tool, you can restrict the build to its packages:
The benchmark packages do not depend on CUDA or on the ZED SDK, so they can also be installed on a remote workstation that only receives the data over the network. This is the best way to measure what a client machine really receives.
Usage
Start the ZED node, then open a new terminal console and run the benchmark node on the topic you want to measure:
For example:
The node prints the benchmark configuration, waits for the topic to be advertised (it polls the graph every 500 msec), and then starts measuring:
The last line is the live statistics line. It is updated in place; the counter is the number of processed messages, and frequency and bandwidth are reported as instant/average. The layout is fixed-width and stays within 80 columns so that it does not wrap on a default terminal.
By default the benchmark runs until you stop it with Ctrl+C. You can also limit it by time or by number of messages, and save the final report to a file.
Run for 30 seconds, then stop and print the report:
Run until 1000 messages are received, then stop:
Run for 60 seconds and also save the report to a log file:
Parameters
All the parameters are read-only: they must be set when the node is started and cannot be modified at runtime.
When both test_duration_sec and test_sample_count are set, the test stops as soon as the first of the two limits is reached.
The value of avg_win_size defines how reactive the average is. A large window (e.g. 500) gives a very stable value, useful to characterize a steady stream; a small window (e.g. 30) reacts quickly and makes throughput drops easier to spot.
Final report
The report is generated when a test limit is reached or when the node is stopped with Ctrl+C. It is printed to the console and, if log_file_path is set, written to that file:
The report contains:
- Topic name and topic type as discovered on the ROS 2 graph.
- Stop reason:
test completed (duration reached),test completed (sample count reached), orinterrupted by the user (Ctrl+C). - Test duration: time between the first and the last received message.
- Messages received and total data received during the test.
- Mean / min / max of frequency, message size, and bandwidth.
If no message is received at all, the report reports it explicitly instead of printing meaningless statistics.
Understanding the statistics
- The instantaneous frequency and bandwidth are computed from the last inter-arrival interval, measured with a monotonic clock (
steady_clock), so the measurement is not affected by host clock adjustments (NTP/PTP). - The average frequency and bandwidth are computed on the message periods and sizes, i.e. as
1/mean(Δt)andmean(size)/mean(Δt). This is the unbiased windowed rate also used byros2 topic hz. Averaging the instantaneous rates directly would systematically overestimate the real frequency, increasingly so on jittery topics. - In the final report, the mean frequency and bandwidth are computed over the whole test (total messages over total time and total data over total time).
- The report min/max are tracked on the windowed average rate, and only after the averaging window has been filled. Tracking them on the raw instantaneous rate would report meaningless extremes caused by a single very short inter-arrival interval, e.g. two messages delivered back-to-back by the executor or a publisher burst. If the test is too short for the window to fill, min and max fall back to the overall mean values.
- The message size is the size of the serialized message and is printed with adaptive units (B / KB / MB / GB), so small messages such as IMU samples are not rounded to
0.00 MB. - Bandwidth values are expressed in Mbps computed on binary multiples (1 Mb = 1024 × 1024 bits).
- The very first received message only initializes the time reference: no frequency can be derived from a single message.
Frequencies measured by the benchmark tool can be lower than the publishing frequencies reported by the node diagnostic. The diagnostic measures the rate at which the node publishes, while the benchmark measures the rate at which a subscriber receives. A difference between the two means that messages are being lost or delayed by the middleware, by the network, or by the receiving machine; see DDS and Network Tuning.
Plotting the statistics
The node publishes the statistics on the topic <name_of_the_benchmarked_topic>_stats, e.g. /zed/zed_node/rgb/color/rect/image_stats, so they can be recorded with ros2 bag or plotted in real time.
The message is a custom type, zed_topic_benchmark_interfaces/msg/BenchmarkStatsStamped, defined as:
For example, to plot the average frequency and bandwidth with rqt_plot:
QoS
The benchmark subscriber uses a Best Effort, KEEP_LAST QoS with a queue depth of 1. A Best Effort subscriber is compatible with both Reliable and Best Effort publishers, so the tool connects out of the box to the sensor-data topics of the ZED nodes (images, depth, point clouds).
If you need to measure the topic with Reliable reliability, you can override the QoS at runtime:
A shallow queue (depth 1) is intentional: it makes the tool measure the data that is actually delivered in time, instead of hiding losses behind a large buffer.
Benchmarking IPC and composition
The zed_topic_benchmark_component package provides the ROS 2 component stereolabs::TopicBenchmarkComponent. Loading it in the same container as the ZED node, with Intra Process Communication enabled, lets you measure the performance of the zero-copy data path and compare it with the standard, DDS-based one.
Start the ZED node with IPC enabled:
then load the benchmark component into the ZED container:
You can verify that the benchmark node is running in the same process as the ZED node with ros2 component list:
Set use_ros_log:=true when the component runs inside a container: the in-place console update of the live statistics line is not usable when the output is multiplexed with the logs of the other nodes.
When the benchmark component reaches a test_duration_sec or test_sample_count limit, it requests the shutdown of the process it is running in. In a shared container this stops the whole container, including the ZED node and any other composed node. The reports are still generated before the shutdown. To benchmark composed nodes for an arbitrary time, leave both limits at their default (infinite) and stop the container with Ctrl+C.
Benchmarking a multi-camera setup
The zed_test_ipc.launch.py launch file of the zed_topic_benchmark package starts a multi-camera setup and one benchmark node per camera, either composed in the camera container using IPC or as separate processes:
The available launch arguments are:
For each camera the launch file benchmarks the topic /zed_multi/<cam_name><topic_name> with a node named benchmark_<index> in the zed_multi namespace. All the benchmark nodes use the ROS logging system (use_ros_log is forced to true); when use_ipc is False the averaging window is also increased to 5000 samples, to obtain very stable average values on long runs.
As explained in the previous section, with use_ipc:=True all the benchmark nodes share the camera container: a finite test_duration_sec/test_sample_count stops the whole container as soon as the first node reaches its limit. Keep the limits at their defaults and stop the launch with Ctrl+C to benchmark a composed multi-camera setup for an arbitrary time.
Testing your ROS 2 configuration
The zed_topic_benchmark package ships a helper script, zed_check_ros2_config.sh, that automates a fixed set of benchmarks against a real ZED camera and prints a report for each one. Its purpose is to verify that your ROS 2, DDS, and system configuration is able to deliver the camera data at the expected rate and bandwidth, a quick way to validate a new setup or to investigate performance problems.
The script benchmarks the following topics, each in both standard mode (the benchmark runs in a separate process, so the data goes through the DDS middleware) and IPC mode (the benchmark is loaded as a component in the same container as the ZED node, with Intra Process Communication):
The RGB image is measured with depth disabled (NONE) to isolate the image throughput, while depth and point cloud are measured with the default depth mode (NEURAL_LIGHT).
For each of the six tests, the script starts the ZED node with the required enable_ipc and depth.depth_mode configuration, waits for the topic to be advertised, lets the node stabilize, runs the measurement, saves the report, and then stops the node to release the camera.
Usage
For example, for a ZED 2i with 20 seconds of measurement per test:
The first argument is the camera model (zed, zedm, zed2, zed2i, zedx, zedxm, …) and defaults to zed2i; the second one is the duration in seconds of each measurement and defaults to 15. The behavior can be further tuned with environment variables:
The script starts and stops the ZED node itself, so no other node must be using the camera when you run it. Make sure that no ZED node is already running and that the camera is free.
The script must be run in a terminal where the ROS 2 and workspace environments are sourced (it checks that the ros2 command is available). Plan for a few minutes of execution: six tests, each including the start-up of the ZED node, the measurement, and the settle time.
Reading the results
A full report is saved for each test in the output directory, together with the log of the ZED node, useful if a test fails or if a topic never appears. At the end, a summary table is printed:
A NO DATA row means that no report was produced for that test: the topic was not published within TOPIC_TIMEOUT, or the ZED node failed to start. Check the corresponding zed_<mode>_<topic>.log file in the output directory.
Compare the measured frequencies with the expected ones, i.e. the values configured through general.pub_frame_rate and depth.point_cloud_freq (see Node Frequency Tuning), and compare the standard rows with the ipc ones to evaluate the gain provided by Composition and IPC.
When the results are not as expected
If the measured frequencies or bandwidths are lower than expected, or if the IPC mode does not improve on the standard mode, your ROS 2 middleware or your system is likely not tuned for high-throughput data. The following pages describe how to fix it:
- DDS and Network Tuning for ROS 2: the DDS and kernel settings required to reliably transport large messages such as images and point clouds.
- Using Zenoh as Middleware: an alternative middleware, recommended for distributed and Wi-Fi based setups.
- Node Frequency Tuning: how to configure the publishing frequency of each module to match the available computational power and bandwidth.
- Composition and IPC: how to remove the middleware overhead entirely when publisher and subscriber run on the same machine.
Testing the packages
The statistics engine, i.e. the sliding-window averages and the frequency/bandwidth math, is covered by C++ unit tests; the end-to-end behavior, i.e. report generation and self-termination on a sample-count limit, is covered by a launch_testing integration test that runs the node against a synthetic publisher.
Always run colcon from the root of your ROS 2 workspace (the folder containing src/, build/, and install/). Running it from another directory makes colcon scan that directory recursively, which can pull in unrelated setup.py files and abort with package-identification errors.
The integration test can also be run directly with launch_test: