Get Started with Isaac™ Sim

NVIDIA® Isaac™ Sim is part of the NVIDIA® Omniverse platform; it is a high-fidelity, photorealistic simulator tailored to robotics simulation. This tutorial covers the essential steps needed to get started with Isaac™ Sim and the ZED. For a more in-depth tutorial, you can follow the official guides from NVIDIA®.

Build and install NVIDIA® Isaac™ Sim

Since version 5.0, the recommended way to install Isaac™ Sim is to build it from source using the official GitHub repository.

Prerequisites

The ZED Camera extension shares the same requirements as Isaac™ Sim itself. Refer to the official Isaac™ Sim requirements page for the full list of hardware and software prerequisites. The two that most often block a first build:

• NVIDIA driver 595.58.03 or newer on Linux (595.97 or newer on Windows). The build does not check the driver, so an unsupported one only shows up when you first launch Isaac™ Sim.
• At least 50 GB of free disk space. A first build consumes roughly 40 GB, and not all of it lands in the clone: the downloaded dependencies and Kit extensions also fill ~/.cache/packman and ~/.local/share/ov in your home directory.

Building from source additionally needs a toolchain that the requirements page does not cover:

Linux:

$
sudo apt-get install build-essential git-lfs gcc-11 g++-11
$
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 200
$
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 200

Windows: install Microsoft Visual Studio 2022 with the Desktop development with C++ workload (which brings the Windows SDK and the MSVC toolchain), plus Git LFS:

1
winget install --id=Microsoft.VisualStudio.2022.Community -e --override "--add Microsoft.VisualStudio.Workload.NativeDesktop --includeRecommended"

Clone and build

$
git clone -b main https://github.com/isaac-sim/IsaacSim.git isaacsim
$
cd isaacsim
$
git lfs install
$
git lfs pull

Linux:

$
./build.sh

Windows:

$
build.bat

Do you accept the governing terms? (yes/No):

Error: NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA Omniverse must be accepted to proceed.

The build downloads tens of gigabytes of dependencies and compiles them, so expect it to run for a while on a first, cold build.

Congrats! 🎉 You are now ready to start using Isaac™ Sim and the ZED Camera extension.

Launch Isaac™ Sim

Once Isaac™ Sim has been built in the previous step, you can run the application from a terminal:

Linux:

$
# From Isaac Sim root directory
$
cd _build/linux-x86_64/release
$
./isaac-sim.sh

Windows:

$
cd _build/windows-x86_64/release
$
isaac-sim.bat

Create your environment

The environment in Isaac™ Sim consists of a Stage (panel to the right) that holds all of the assets in the USD scene. To have a working simulation with physics, you need to:

• Create a physics scene that enables physics simulation and gravity.
• Add a ground plane that prevents objects from falling infinitely.

A quick way to explore these things is to use a predefined environment by going to the top menu bar Create -> Isaac -> Environments and choosing the Flat grid as it is the simplest one.

You will notice that the stage has been populated with items represented by different prims (Prim is short for primitive, the fundamental container object in Omniverse). The window that shows the flat grid is called the Viewport.

Add objects to the scene

You can use the Content and Isaac Assets panels located at the bottom of the window to add prims to the stage.

The Content tab lets you look for files located on your machine, while the Isaac Assets tab shows all of the available assets in Isaac™ Sim. To add an item, simply drag it into your scene or the stage panel.

Create an Action Graph

Action Graphs are part of the Visual Scripting tools provided by Isaac™ Sim, allowing for programming using nodes instead of code. They are event-based and enable the user to execute actions based on a sequence of events and conditions. To create an Action Graph, right-click anywhere in the viewport and select Create -> Visual Scripting -> Action Graph.

A new Action Graph appears in the Stage view, and the related Action Graph panel is now open at the bottom of the screen.

Start typing “OnPlaybackTick” in the search bar on the left side of the panel to find the node. Then drag and drop it into the graph. Do the same for the “Print Text” node.

Connect the Tick of the OnPlaybackTick to the In of the Print Text. Then click on the Print Text node and fill out its properties in the Property panel on the right.

You can experiment with different settings to print on the screen (viewport) or on the Console tab. To execute the graph, simply press play (to trigger a playback tick) and you should see the printed text.

For a detailed explanation of OmniGraph, Omniverse’s visual programming framework, refer to the official documentation, which shows you how to control a robot using Action Graphs.

Install the ZED SDK

To receive the simulated camera stream, the ZED SDK 5.4.1 or newer must be installed on the machine that runs your application. This does not have to be the machine running Isaac™ Sim: the stream can be received over the network, or on the same machine using low-latency IPC. You can find the installation instructions here.

Install the ZED Camera extension

The ZED Camera extension is not published in the Isaac™ Sim extension registry, so it is not listed in the extension manager until you point Isaac™ Sim at a local copy. Clone and build the GitHub repository:

$
cd <your_workspace>
$
git clone https://github.com/stereolabs/zed-isaac-sim.git
$
cd zed-isaac-sim

Linux:

$
./build.sh

Windows:

$
build.bat

The build downloads the runtime ZED library (libsl_zed) and the ZED Sim2Real library from the Stereolabs CDN, then compiles the C++ plugin. If the optional Sim2Real archive cannot be downloaded, the build still succeeds and prints a warning; only the ZED Sim2Real feature is disabled.

Now register the extension with Isaac™ Sim. Open the Extensions view from the menu, Window -> Extensions. Click the menu button (hamburger icon near the search field), open Settings, and click the + button to add the path to the exts folder of the cloned repository (e.g. <your_workspace>/zed-isaac-sim/exts).

Enable the ZED Camera extension in the Third-Party tab.

Verify the installation (headless)

Before building a scene in the GUI, you can run a quick headless check to confirm that Isaac™ Sim starts and that the ZED Camera extension and its OmniGraph nodes load correctly. This is handy for automated or CI setups.

Save the following script as verify_zed_extension.py, updating EXTS_FOLDER to point to the exts folder of your cloned extension:

1
from isaacsim import SimulationApp
2
3
# Start Isaac Sim with no GUI window
4
sim = SimulationApp({"headless": True})
5
6
import omni.kit.app
7
import omni.graph.core as og
8
9
EXT_ID = "sl.sensor.camera"
10
EXTS_FOLDER = "/path/to/zed-isaac-sim/exts"  # folder that contains sl.sensor.camera
11
12
# Enable the ZED Camera extension from its local path
13
manager = omni.kit.app.get_app().get_extension_manager()
14
manager.add_path(EXTS_FOLDER)
15
manager.set_extension_enabled_immediate(EXT_ID, True)
16
for _ in range(20):
17
    sim.update()
18
19
results = ["extension_enabled=%s" % manager.is_extension_enabled(EXT_ID)]
20
21
# Instantiate each ZED node type to confirm it is registered
22
# (og.Controller.edit raises if a node type is missing).
23
keys = og.Controller.Keys
24
node_types = [
25
    "sl.sensor.camera.ZED_Camera",           # ZED Camera Helper (stereo)
26
    "sl.sensor.camera.ZED_Camera_One",       # ZED Camera One Helper (mono)
27
    "sl.sensor.camera.OgnZEDSimCameraNode",  # ZED Stream (low-level)
28
]
29
for i, node_type in enumerate(node_types):
30
    og.Controller.edit(
31
        {"graph_path": "/World/Check_%d" % i, "evaluator_name": "execution"},
32
        {keys.CREATE_NODES: [("n%d" % i, node_type)]},
33
    )
34
    results.append("registered: %s" % node_type)
35
36
# Write the outcome to a file: Isaac Sim's close() can drop buffered stdout.
37
with open("zed_verify_result.txt", "w") as f:
38
    f.write("\n".join(results) + "\nZED_EXTENSION_OK\n")
39
40
sim.close()

Run it with the Python interpreter shipped with your Isaac™ Sim build:

$
# From the Isaac Sim build output directory
$
cd <isaacsim>/_build/linux-x86_64/release
$
./python.sh /path/to/verify_zed_extension.py
$
$
cat zed_verify_result.txt

If everything is set up correctly, zed_verify_result.txt ends with ZED_EXTENSION_OK and lists the three registered node types (ZED Camera Helper, ZED Camera One Helper, and the low-level ZED Stream). A missing node type makes the script exit with an error instead.

Now that we have a solid grasp of Isaac™ Sim’s core concepts and interface, and the ZED Camera extension installed, we are ready to configure a ZED camera within the Isaac™ Sim environment.