Software IntegrationIsaac Sim

SDK-free depth capture in Isaac™ Sim

Open in ClaudeOpen in ChatGPT

The ZED Camera extension can capture RGB and depth from a simulated ZED directly inside NVIDIA® Isaac™ Sim, using the simulator’s renderer, with no ZED SDK and no streaming. This is useful for quick frame/depth capture, generating datasets, or working on a machine where the ZED SDK is not installed.

For how this compares to the streaming and Isaac Lab tensor paths, see Choosing a data path.

The depth captured here is Isaac™ Sim’s ground-truth renderer depth (distance_to_image_plane), not the ZED SDK’s stereo-matched depth. For genuine ZED SDK depth, use the streaming path.

ZED Depth OmniGraph node

The extension provides a ZED Depth node that captures RGB + depth each simulation tick. Add it to an Action Graph, connect an On Playback Tick node to its ExecIn, set its properties, and press Play. The ZED Depth node supports stereo camera models only; mono ZED X One cameras are not supported.

PropertyDescription
ZED Camera PrimThe ZED camera prim in the stage (e.g. /World/ZED_X).
Camera ModelSimulated stereo model: ZED_X, ZED_XM, ZED_X_Nano, ZED_M, or ZED_2i. Detected automatically from the connected ZED Camera Prim when possible.
Lens TypeWide or Narrow (options depend on the model). Detected automatically like the Camera Model.
ResolutionRender resolution: HD2K, HD1200, HD1080, HD720, SVGA, or VGA (filtered per model).
Enable SaveSave RGB (PNG) + depth (EXR) to disk each frame.
Output DirectoryDirectory to save frames. Required when Enable Save is on.

Standalone scripts

The extension also ships two capture scripts under exts/sl.sensor.camera/examples/isaac_sim/, both driven by the sl.sensor.camera.zed_depth.ZEDDepthCamera helper.

Standalone / headless (zed_depth_standalone.py)

Captures left RGB + depth for N frames, prints per-frame stats, and optionally saves PNG/EXR. It runs with no arguments: when no scene or camera is given, it builds a minimal ground + dome-light scene and spawns a ZED camera at a small default pose, so you get a usable depth capture out of the box.

The script is launched with the Python interpreter shipped with Isaac™ Sim (python.sh / python.bat, in the _build/linux-x86_64/release folder of your Isaac™ Sim build). That interpreter and the example live in two different repositories, so run python.sh from the Isaac™ Sim build directory and give the script an absolute path into your zed-isaac-sim clone:

$# Linux, zero-config: minimal scene + auto-spawned ZED_X
$cd <isaacsim>/_build/linux-x86_64/release
$./python.sh <your_workspace>/zed-isaac-sim/exts/sl.sensor.camera/examples/isaac_sim/zed_depth_standalone.py
$
$# Full control: load your own scene and/or use a camera already placed in it
$./python.sh <your_workspace>/zed-isaac-sim/exts/sl.sensor.camera/examples/isaac_sim/zed_depth_standalone.py \
>    --usd_path /path/to/scene.usd \
>    --camera_prim /World/ZED_X \
>    --camera_model ZED_X \
>    --resolution HD1200 \
>    --num_frames 10 \
>    --output_dir /tmp/zed_depth
$# Windows: use python.bat. Add --headless for no GUI.

Arguments (all optional): --usd_path (environment USD; omit to build a minimal scene), --camera_prim (an existing ZED camera prim; omit to spawn one), --camera_model, --resolution, --num_frames, --output_dir, --headless.

Saving depth needs the OpenEXR codec, which is disabled by default in the OpenCV build shipped with Isaac™ Sim. Whenever you pass --output_dir (or enable Enable Save on the ZED Depth node), set OPENCV_IO_ENABLE_OPENEXR=1 before launching, or the run aborts on the first frame with:

cv2.error: OpenCV(4.13.0) ... (-213:The function/feature is not implemented)
imgcodecs: OpenEXR codec is disabled. You can enable it via 'OPENCV_IO_ENABLE_OPENEXR' option.
$# Linux
$OPENCV_IO_ENABLE_OPENEXR=1 ./python.sh <...>/zed_depth_standalone.py --output_dir /tmp/zed_depth
1# Windows
2set OPENCV_IO_ENABLE_OPENEXR=1
3python.bat <...>\zed_depth_standalone.py --output_dir C:\zed_depth

Capturing without --output_dir is unaffected: the depth arrays are produced normally and only the EXR write needs the codec.

GUI Script Editor (zed_depth_example.py)

For interactive use inside the Isaac™ Sim GUI (extension enabled): load a scene with a ZED camera, paste the script into the Script Editor, edit the CONFIG block (CAMERA_PRIM_PATH, CAMERA_MODEL, RESOLUTION, CAPTURE_INTERVAL, SAVE_DIR), execute it, then press Play. RGB + depth are captured every CAPTURE_INTERVAL physics steps; press Stop to clean up.

Output

  • Console: per-capture shape and depth-range stats.
  • Files (when a save path is set): rgb_left_NNNNNN.png (left RGB) and depth_NNNNNN.exr (float32 depth in meters, 0 = invalid). For stereo models, rgb_right_NNNNNN.png (right RGB) is saved too; depth stays left-referenced.

See also