Setting up ZED Camera in Isaac Sim

This tutorial guides you through setting up the ZED X camera in NVIDIA® Isaac Sim and connecting to the ZED SDK. If you’re not familiar with Isaac Sim, make sure to go through the Isaac Sim Official documentation.

Setting up your virtual camera #

Add the model to the stage #

The ZED X model is available in the Isaac Assets tab on the bottom panel.

If you cannot find the ZED X Model, then you can manually add it by cloning the GitHub repository:

cd <your_workspace>
git clone git@github.com:stereolabs/zed-isaac-sim.git

Select the model ZED_X.usd and drag it in the Viewport or in the Stage view.

You will notice the prim has a base_link XForm, this is the reference frame of the mounting point on the bottom of the camera.

Activate the ZED Camera extension #

The virtual camera takes advantage of the ZED SDK streaming input to stream its data into the ZED SDK. The first step is to enable the ZED Camera extension to get access to the ZED Camera Streamer Omnigraph node.
In the top menu bar, navigate to Window -> Extension to show the extension manager.

If you cannot find the ZED X Extension, then you can manually add it by cloning the GitHub repository, if you have not yet done this before:

cd <your_workspace>
git clone git@github.com:stereolabs/zed-isaac-sim.git

Click on the menu button (three horizontal bars), select Settings, and create a new entry in the right list with the path to the root folder of the cloned repository.

Now search for zed in the search bar on the left column, and hit INSTALL to install the ZED Extension.

Finally, toggle the DISABLED switch to activate the extension and flag the AUTOLOAD option if you want the extension to be loaded when the simulator starts.

Build the action graph #

To initiate the virtual ZED camera streaming, we will be using the ZED Camera streamer Omnigraph node. To stream data at each simulation tick, drag and drop this node into the graph (you can type zed in the search panel to quickly find it), add its ZED Camera prim target to the camera in the scene, and connect its ExecIn input to the Tick output of the On Playback Tick node.

You can also configure the streamer properties in the Property panel on the right:

• Serial number: this field allows you to identify the virtual camera you are using. It has no functional effect and can be left to its default value. This field can be useful when using multiple cameras to differentiate them.
• Streaming port: Defines which port is used for streaming. It must be an even number.
• Use system time: when set to true, the streamer assigns the latest system time to the images that are sent to the SDK. If set to false, the streamer only sets the first timestamp to system time and then increments the next timestamps by simulation time.
• ZED Camera prim: this parameter should be set to the ZED X camera prim in the stage as shown in the picture above.

And that’s it, you can now start the simulation to stream your stage into the ZED SDK!

Test it in ZED DepthViewer #

Once streaming is started, you can test it by connecting the ZED DepthViewer to the virtual camera. Simply launch the tool, click on the streaming input button, set the IP Address (the streaming computer) and the port (set in Streaming port), and you should see the depth data coming from the virtual environment.