Software IntegrationIsaac ROS

Setting up Isaac™ ROS to work with ZED Cameras in ROS 2

Open in ClaudeOpen in ChatGPT

This guide will help you set up the NVIDIA® Isaac™ ROS packages to work with the ZED ROS 2 Wrapper.

A solid connection to the internet is required to download the NVIDIA® Isaac™ ROS packages and the StereoLabs software If you are using a Jetson device, we recommend using a wired connection for better stability and speed.

NVIDIA® Jetson™ Setup

you can ignore this section if you are not using an NVIDIA® Jetson™ device.

Before installing Isaac™ ROS on an NVIDIA® Jetson™ device, ensure that it is set up correctly and has the necessary software installed.

Install the prerequisites

Open a terminal [Ctrl+Alt+T] and enter the following commands to install the prerequisites:

$sudo apt-get update
$sudo apt-get install python3 python3-pip # Install python3
$sudo apt-get install nvidia-jetpack # Install the Jetson developer tools
$sudo pip3 install jetson-stats # Optional but recommended to monitor the status of the Jetson™ module

Install the VPI library

Configure the VPI library for running compute on Jetson™ device’s PVA accelerator.

  1. Generate CDI Spec for GPU/PVA:

    $sudo nvidia-ctk cdi generate --mode=csv --output=/etc/cdi/nvidia.yaml

  2. Install pva-allow-2 package:

    $# Add Jetson public APT repository
    $sudo apt-get update
    $sudo apt-get install software-properties-common
    $sudo apt-key adv --fetch-key https://repo.download.nvidia.com/jetson/jetson-ota-public.asc
    $sudo add-apt-repository 'deb https://repo.download.nvidia.com/jetson/common r36.4 main'
    $sudo apt-get update
    $sudo apt-get install -y pva-allow-2

Install an additional SSD memory if you are using an NVIDIA® Jetson™ development kit

Using Isaac™ ROS with Docker requires a fast and large storage solution, such as an SSD, to ensure optimal performance.

The Docker images can be quite large, and running them on slower storage devices may lead to performance issues.

If you are using an NVIDIA® Jetson™ development kit device, we recommend you follow the official NVIDIA® documentation for best practices on setting up a NVMe SSD card in the Jetson developer kit’s carrier board.

If you are using instead a StereoLabs ZED Box, ZED Box Mini, or a Mini Carrier board, the preinstalled NVMe SSD is already configured for optimal performance and no further configuration is required.

Configure NVIDIA® Isaac™ ROS using Docker

You can skip this section if you are not using Docker to run your Isaac™ ROS applications and prefer to use the official Apt Repository.

This setup is recommended when using an NVIDIA® Jetson™ device.

NVIDIA® provides ready-to-use Docker images for using Isaac™ ROS. The Docker configuration allows you to easily set up your development environment with all the necessary dependencies and tools pre-installed on Jetson devices and x86_64 systems.

Please refer to the official NVIDIA® Isaac™ ROS documentation for advanced configurations and usage.

  1. Install the pre-requisite:

    $sudo apt-get install python3-pip # Install python3

  2. Restart Docker:

    $sudo systemctl daemon-reload && sudo systemctl restart docker

  3. Install Git LFS to pull down all large files:

    $sudo apt-get install git-lfs
    $git lfs install  --skip-repo

  4. Create a workspace and add it to the environment:

    $mkdir -p ~/ros2/isaac_ros_ws/src
    $echo "export ISAAC_ROS_WS=${HOME}/ros2/isaac_ros_ws/" >> ~/.bashrc
    $source ~/.bashrc

  5. Enter the workspace and clone packages:

    $cd ${ISAAC_ROS_WS}/src
    $git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git --branch release-3.2 --recurse-submodules

  6. Pull the official Isaac Common Docker image and start the first instance of development environment:

    $cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
    >./scripts/run_dev.sh

  7. Wait for the build to complete. This may take some time ☕

  8. When the build is complete, the Docker container will start automatically. You can verify that you are inside the Docker container by checking the command prompt, which should look like admin@ubuntu:/workspaces/isaac_ros-dev$.

    Enter the command exit to leave the Docker container.

For any problems with the installation please refer to the NVIDIA® Isaac™ ROS Troubleshooting documentation.

Configure NVIDIA® Isaac™ ROS using Docker with ZED cameras

To use Isaac™ ROS with ZED cameras, you must create a Docker layer that installs the ZED SDK in the Isaac™ ROS Docker image. This layer will allow you to use ZED cameras in your Isaac™ ROS applications.

  1. Create a new entrypoint script to change the permissions of the ZED SDK folder when the container starts to allow access to non-root users:

    $cd ${ISAAC_ROS_WS}/src/isaac_ros_common/docker/scripts
    $touch zed-entrypoint.sh
    $chmod +x zed-entrypoint.sh

  2. Open the zed-entrypoint.sh file in a text editor and add the following content:

    $#!/bin/bash
    $
    $# Enable ZED if installed
    $echo "Checking ZED SDK..."
    $
    $ZED_FOLDER=/usr/local/zed/
    $
    $if [ -d "${ZED_FOLDER}" ]; then
    $   echo " * ${ZED_FOLDER} exists. Changing permissions"
    $   chown -R ${USERNAME}:${USERNAME} ${ZED_FOLDER}
    $   echo " * ZED SDK available:"
    $   ${ZED_FOLDER}tools/ZED_Explorer -v
    $else
    $   echo " * $ZED_FOLDER does not exist. ZED SDK not installed"
    $   exit 1
    $fi

  3. Create a new Dockerfile.zed file in the isaac_ros_common/docker directory:

    $cd ${ISAAC_ROS_WS}/src/isaac_ros_common/docker
    $touch Dockerfile.zed

  4. Open the Dockerfile.zed file in a text editor and add the following content:

    1ARG BASE_IMAGE=NONE
    2FROM ${BASE_IMAGE} AS catscanners
    3
    4# disable terminal interaction for apt
    5ENV DEBIAN_FRONTEND=noninteractive
    6ENV SHELL=/bin/bash
    7SHELL ["/bin/bash", "-c"]
    8
    9ARG ARCH=aarch64
    10#ARG ARCH=x86_64
    11
    12# Copy the SDK install script
    13COPY scripts/install-zed-${ARCH}.sh /opt/zed/install-zed.sh
    14
    15# Copy the entrypoint script to change the permissions of the ZED SDK folder
    16# when the container starts to the correct location
    17COPY scripts/zed-entrypoint.sh /usr/local/bin/scripts/entrypoint_additions/zed-entrypoint.sh
    18
    19# Install the ZED SDK
    20RUN chmod +x /opt/zed/install-zed.sh && \
    21   /opt/zed/install-zed.sh

    Uncomment the ARG ARCH=x86_64 line if you are using an x86_64 system instead of a Jetson™ device.

    The Dockerfile copies the zed-entrypoint.sh script to the /usr/local/bin/scripts/entrypoint_additions/ directory. This will ensure that the script is executed when the Docker container starts, as the default entrypoint script in the Isaac™ ROS Docker image is designed to run any scripts found in this directory at startup.

  5. Modify the install-zed-<arch>.sh file to install the latest ZED SDK.

    A. Open the isaac_ros_common/docker/scripts/install-zed-<arch>.sh file in a text editor. Replace <arch> with aarch64 if you are using a Jetson device or x86_64 if you are using an x86_64 system.

    B. Replace the content of the file with the following script:

    Jetson:

    click the arrow to expand the full script content

    scripts/install-zed-aarch64.sh
    $#!/bin/bash
    $
    $# Download dependencies for zed SDK installation RUN file
    $sudo apt-get update -y || true
    $sudo apt-get install --no-install-recommends lsb-release wget less zstd udev sudo apt-transport-https -y
    $
    $# Download zed SDK installation RUN file to /tmp directory
    $cd /tmp
    $
    $URL=https://download.stereolabs.com/zedsdk/5.0/l4t36.4/jetsons
    $wget -q --no-check-certificate -O ZED_SDK_Linux.run ${URL}
    $SIZE=$(du -sb './ZED_SDK_Linux.run' | awk '{ print $1 }')
    $if ((SIZE<1024)) ; then
    $   echo "ERROR: ZED_SDK_Linux.run size is not valid: ${SIZE}B!!!";
    $   echo " * Verify the ZED SDK download link validity: " ${URL} ;
    $   exit 1
    $fi
    $
    $sudo chmod 777 ./ZED_SDK_Linux.run
    $sudo ./ZED_SDK_Linux.run silent skip_od_module skip_python skip_drivers
    $# Symlink required to use the streaming features on Jetson inside a container, based on
    $# https://github.com/stereolabs/zed-docker/blob/fd514606174d8bb09f21a229f1099205b284ecb6/4.X/l4t/devel/Dockerfile#L27C5-L27C95
    $sudo ln -sf /usr/lib/aarch64-linux-gnu/tegra/libv4l2.so.0 /usr/lib/aarch64-linux-gnu/libv4l2.so
    $
    $# Install zed-ros2-wrapper dependencies
    $sudo apt-get update && sudo apt-get install -y \
    >   ros-humble-zed-msgs \
    >   ros-humble-nmea-msgs \
    >   ros-humble-geographic-msgs \
    >   ros-humble-robot-localization \
    >   #ros-humble-point-cloud-transport \
    >   #ros-humble-point-cloud-transport-plugins \
    >   #ros-humble-draco-point-cloud-transport \
    >   #ros-humble-zlib-point-cloud-transport \
    >   #ros-humble-zstd-point-cloud-transport \
    >   #ros-humble-point-cloud-transport \
    >
    $# Uncomment the point cloud transport packages above if you want to use point cloud compression.
    $
    $# Install base Isaac ROS Nitros packages
    $sudo apt-get update && sudo apt-get install -y \
    >   ros-humble-isaac-ros-common \
    >   ros-humble-isaac-ros-nitros \
    >   ros-humble-isaac-ros-managed-nitros \
    >   ros-humble-isaac-ros-nitros-image-type
    $
    $# Cleanup
    $sudo rm -rf /usr/local/zed/resources/*
    $rm -rf ZED_SDK_Linux.run
    $sudo rm -rf /var/lib/apt/lists/*

    • If you are using a different version of the JetPack SDK, you must adjust the URL accordingly. For example, if you are using JetPack 6.0, you must change l4t36.4 to l4t36.3.

    • If you are using a different version of the ZED SDK, you must adjust the URL accordingly. For example, if you want to install ZED SDK 5.1, you must change 5.0 to 5.1.

    x86_64:

    click the arrow to expand the full script content

    scripts/install-zed-x86_64.sh
    $#!/bin/bash
    $
    $# Download dependencies for zed SDK installation RUN file
    $sudo apt-get update -y || true
    $sudo apt-get install --no-install-recommends lsb-release wget less udev sudo zstd build-essential cmake libpng-dev libgomp1 -y
    $
    $# Download zed SDK installation RUN file to /tmp directory
    $cd /tmp
    $URL=https://download.stereolabs.com/zedsdk/5.0/cu12/ubuntu24
    $wget -q --no-check-certificate -O ZED_SDK_Linux.run ${URL}
    $SIZE=$(du -sb './ZED_SDK_Linux.run' | awk '{ print $1 }')
    $if ((SIZE<1024)) ; then
    $   echo "ERROR: ZED_SDK_Linux.run size is not valid: ${SIZE}B!!!";
    $   echo " * Verify the ZED SDK download link validity: " ${URL} ;
    $   exit 1
    $fi
    $
    $sudo chmod 777 ./ZED_SDK_Linux.run
    $sudo ./ZED_SDK_Linux.run silent skip_od_module skip_python skip_drivers
    $
    $
    $# Install zed-ros2-wrapper dependencies
    $sudo apt-get update && sudo apt-get install -y \
    >   ros-humble-zed-msgs \
    >   ros-humble-nmea-msgs \
    >   ros-humble-geographic-msgs \
    >   ros-humble-robot-localization \
    >   #ros-humble-point-cloud-transport \
    >   #ros-humble-point-cloud-transport-plugins \
    >   #ros-humble-draco-point-cloud-transport \
    >   #ros-humble-zlib-point-cloud-transport \
    >   #ros-humble-zstd-point-cloud-transport \
    >   #ros-humble-point-cloud-transport \
    >
    $# Uncomment the point cloud transport packages above if you want to use point cloud compression.
    $
    $# Install base Isaac ROS Nitros packages
    $sudo apt-get update && sudo apt-get install -y \
    >   ros-humble-isaac-ros-common \
    >   ros-humble-isaac-ros-nitros \
    >   ros-humble-isaac-ros-managed-nitros \
    >   ros-humble-isaac-ros-nitros-image-type
    $
    $# Cleanup
    $sudo rm -rf /usr/local/zed/resources/*
    $rm -rf ZED_SDK_Linux.run
    $sudo rm -rf /var/lib/apt/lists/*

    • If you are using a different version of the ZED SDK, you must adjust the URL accordingly. For example, if you want to install ZED SDK 5.1, you must change 5.0 to 5.1.

    • If you are using a different version of CUDA, you must adjust the URL accordingly. For example, if you want to install the ZED SDK for CUDA 13, you must change cu12 to cu13.

    • If you are using a different version of Ubuntu, you must adjust the URL accordingly. For example, if you want to install the ZED SDK for Ubuntu 22.04, you must change ubuntu24 to ubuntu22.

  6. Build the Docker image with the ZED SDK installed:

$cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
>./scripts/run_dev.sh -i ros2_humble.zed \
>-a "-v /usr/local/zed/settings:/usr/local/zed/settings \
>    -v /usr/local/zed/resources:/usr/local/zed/resources"

Wait for the build to complete. This may take some time ☕

  1. When the build is complete, the Docker container will start automatically. You can check that the ZED SDK is installed by running the following commands inside the Docker container:

    Check the ZED SDK version:

    $/usr/local/zed/tools/ZED_Explorer --version

    Check camera opening and Depth extraction capabilities:

    $/usr/local/zed/tools/ZED_Depth_Viewer

Install the ZED ROS 2 Wrapper packages

Start a new terminal and enter the Docker container with the ZED SDK installed:

$cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
>./scripts/run_dev.sh -i ros2_humble.zed \
>-a "-v /usr/local/zed/settings:/usr/local/zed/settings \
>    -v /usr/local/zed/resources:/usr/local/zed/resources"

You can now install the ZED ROS 2 Wrapper packages with NITROS support to be used in your Isaac™ ROS applications:

$# Move to the `src` folder of the ROS 2 Workspace
$cd ${ISAAC_ROS_WS}/src
$git clone https://github.com/stereolabs/zed-ros2-wrapper.git
$cd ..
$sudo apt-get update
$# Install the required dependencies
$rosdep install --from-paths src --ignore-src -r -y
$# Build the wrapper
$colcon build --symlink-install --cmake-args=-DCMAKE_BUILD_TYPE=Release --packages-up-to zed_wrapper
$# Setup the environment variables
$echo source $(pwd)/install/local_setup.bash >> ~/.bashrc
$source ~/.bashrc

If you are new to the ZED ROS 2 Wrapper, we recommend starting with the Getting Started with ROS 2 and ZED guide. This guide will help you set up your environment and run your first ZED camera application using ROS 2.

You can also explore the ZED ROS 2 Examples repository for tutorials and advanced examples.

If the ZED ROS 2 Wrapper was already installed in your workspace, just clean and rebuild your workspace to ensure that the ZED ROS 2 Wrapper is built with the NVIDIA® Isaac™ ROS support.

$cd ${ISAAC_ROS_WS}
$rm -rf build/ install/ log/
$colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release --packages-up-to zed_wrapper
$source install/local_setup.bash

You can verify that the ZED ROS 2 Wrapper is built with NVIDIA® Isaac™ ROS support by checking the output of the colcon build command. You should see messages indicating that the isaac_ros_nitros package is being used:

1--- stderr: zed_components
2 * ISAAC ROS Nitros transport is available
3---

Configure NVIDIA® Isaac™ ROS from the official Apt Repository

you can skip this section if you are using Docker to run your Isaac™ ROS applications.

On Desktop machines, you can easily install the NVIDIA® Isaac™ ROS packages using the official Apt Repository. This repository provides pre-built packages for all supported platforms, making it easy to get started with Isaac™ ROS without using Docker.

We do not recommend this procedure when working with Jetson devices because it’s not straightforward to satisfy all dependencies.

NVIDIA® Isaac™ Apt Repository built on NVIDIA’s CDN (Content Distribution Network) hosts the following categories of packages for all users.

  • All Isaac™ ROS packages pre-built for the currently supported platforms.
  • Auxiliary packages to improve developer experience or to support a legacy OS.

Set up the Isaac™ Apt Repository

Open a terminal and follow these steps to set up the NVIDIA® Isaac™ ROS Apt Repository on your system. This guide assumes you are using Ubuntu 22.04.

  1. Set Locale - Ensure your system locale is set to UTF-8:

    $locale  # check for UTF-8
    $
    $sudo apt-get update && sudo apt-get install locales
    $sudo locale-gen en_US en_US.UTF-8
    $sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
    $export LANG=en_US.UTF-8
    $
    $locale  # verify settings

  2. Install Dependencies - Install the required dependencies for the Isaac™ Apt Repository:

    $sudo apt-get update && sudo apt-get install gnupg wget
    $sudo apt-get install software-properties-common
    $sudo add-apt-repository universe

  3. Setup Source - Register the GPG key with apt. Two options are listed below, one for .com (US CDN) and one for .cn (China CDN) - choose whichever you prefer.

    $wget -qO - https://isaac.download.nvidia.com/isaac-ros/repos.key | sudo apt-key add -
    $grep -qxF "deb https://isaac.download.nvidia.com/isaac-ros/release-3 $(lsb_release -cs) release-3.0" /etc/apt/sources.list || \
    >echo "deb https://isaac.download.nvidia.com/isaac-ros/release-3 $(lsb_release -cs) release-3.0" | sudo tee -a /etc/apt/sources.list
    $sudo apt-get update

  4. Add the NVIDIA Keyring - Register the NVIDIA GPG key with apt:

    $sudo apt-get update && sudo apt-get install curl -y
    $sudo curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg
    $echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null

You are now ready to install the NVIDIA® Isaac™ ROS packages.

Install the required NVIDIA® Isaac™ ROS packages

Open a terminal and run the following command to install the required NVIDIA® Isaac™ ROS packages:

$sudo apt-get update
$sudo apt-get install ros-humble-isaac-ros-managed-nitros ros-humble-isaac-ros-nitros-image-type

Install the ZED SDK and ZED X Driver

If you haven’t already, install the ZED SDK and the ZED X Driver on your system. The ZED X Driver is required to use ZED X, ZED X Mini, and ZED X One cameras with Isaac™ ROS.

Install the ZED ROS 2 Wrapper

You can now install the ZED ROS 2 Wrapper packages with NITROS support to be used in your Isaac™ ROS applications by following the instructions in the ZED ROS 2 Wrapper Installation Guide. The ZED ROS 2 Wrapper will automatically detect the available NVIDIA® Isaac™ ROS packages and enable NITROS support during the build process.

If you are new to the ZED ROS 2 Wrapper, we recommend starting with the Getting Started with ROS 2 and ZED guide. This guide will help you set up your environment and run your first ZED camera application using ROS 2.

You can also explore the ZED ROS 2 Examples repository for tutorials and advanced examples.

If the ZED ROS 2 Wrapper was already installed in your workspace, just clean and rebuild your workspace to ensure that the ZED ROS 2 Wrapper is built with the NVIDIA® Isaac™ ROS support.

$cd ~/your_workspace # e.g. cd ~/devel/ros_ws
$rm -rf build/ install/ log/
$colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release --packages-up-to zed_wrapper
$source install/local_setup.bash

You can verify that the ZED ROS 2 Wrapper is built with NVIDIA® Isaac™ ROS support by checking the output of the colcon build command. You should see messages indicating that the isaac_ros_nitros package is being used:

1--- stderr: zed_components
2 * ISAAC ROS Nitros transport is available
3---

Validate the installation

To validate that the installation was successful and that the ZED ROS 2 Wrapper is properly configured with NVIDIA® Isaac™ ROS support, you can run the following command:

$ros2 launch zed_wrapper zed_camera.launch.py camera_model:=<your_camera_model> enable_ipc:=false

If the node starts successfully without errors, your installation is likely correct. You can also check the ROS 2 topic list to see if the ZED camera topics are being published:

$ros2 topic list | grep zed

You should see topics related to the ZED camera, such as /zed/zed_node/rgb/color/rect/image/nitros and /zed/zed_node/depth/depth_registered/nitros, indicating that the ZED ROS 2 Wrapper is running with NITROS support as expected.

When using the ZED ROS 2 Wrapper with NVIDIA® Isaac™ ROS, make sure to set the enable_ipc parameter to false in the launch file to avoid potential conflicts with the NITROS transport. Read more in the ZED Isaac™ ROS Troubleshooting Guide and the NVIDIA® Isaac™ ROS Troubleshooting Documentation.

Good practices

When starting the Isaac™ ROS Docker container, it’s a good practice to enable a few shared volumes to persist your data and configurations across sessions.

Here are some recommended volume mounts:

  • /usr/local/zed/settings:/usr/local/zed/settings if you plan to use the robot in an Internet-negated area, and you previously downloaded the camera calibration files by following this guide.
  • /usr/local/zed/resources:/usr/local/zed/resources if you plan to use the AI module of the ZED SDK (Object Detection, Skeleton Tracking, NEURAL depth) we suggest binding mounting a folder to avoid downloading and optimizing the AI models each time the Docker image is restarted. The first time you use an AI model inside the Docker image, it will be downloaded and optimized in the local bound-mounted folder, and stored there for the next runs.

To use these volume mounts, use the following command when starting the Docker container:

$cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
>./scripts/run_dev.sh -i ros2_humble.zed \
>-a "-v /usr/local/zed/settings:/usr/local/zed/settings \
>    -v /usr/local/zed/resources:/usr/local/zed/resources"