Using the ZED in Isaac™ Lab

Isaac™ Lab is NVIDIA®‘s framework for robot learning built on top of Isaac™ Sim. The ZED Camera extension lets you use a simulated ZED camera directly inside Isaac™ Lab and read its stereo RGB pair and a depth map as in-process tensors, with no ZED SDK and no network streaming.

This is the path for reinforcement learning and imitation learning workflows, where you want camera observations as tensors on the GPU rather than a ZED SDK stream. For how it compares to the streaming and SDK-free depth paths, see Choosing a data path.

The depth here is Isaac™ Sim’s simulated renderer depth (distance_to_image_plane), not the ZED SDK’s stereo-matched depth. For genuine ZED SDK depth, use the streaming path.

How it works

The ZED USD model (e.g. ZED_X.usdc) has authored CameraLeft / CameraRight prims with the correct intrinsics and stereo baseline. In Isaac™ Lab you reference that USD, attach an isaaclab.sensors.Camera sensor to each of those prims, and read rgb and distance_to_image_plane each simulation step. To place or mount the camera, you move the whole USD root so the stereo rig moves rigidly with it; you never transform the individual camera prims.

Requirements

An Isaac™ Lab installation (verified on Isaac™ Lab 3.0 / Kit 110.1.2).
The ZED Camera extension (from the zed-isaac-sim repository) on the Python path. The shipped examples handle this automatically via a small bootstrap import.
Launch through the Isaac™ Lab launcher (isaaclab.bat / isaaclab.sh). The shipped examples enable cameras automatically; when launching your own scripts, pass --enable_cameras: RGB and depth are render-based, so without it the renderer is off and frames are empty.

The first rendered frame compiles RTX shaders and can take several minutes on a cold cache. It is compiling, not hung; subsequent runs are fast.

Quick start

The extension ships ready-to-run examples under exts/sl.sensor.camera/examples/isaaclab/. To add a single ZED X to a scene and read its stereo RGB + depth tensors:

$ # Windows
$ isaaclab.bat -p exts/sl.sensor.camera/examples/isaaclab/zed_single.py --resolution SVGA --save_dir C:\zed_lab_out
$ 
$ # Linux
$ ./isaaclab.sh -p exts/sl.sensor.camera/examples/isaaclab/zed_single.py --resolution SVGA --save_dir /tmp/zed_lab_out

Other examples include a vectorized multi-environment capture (zed_multi_env.py) and ports of flagship Isaac™ Lab demos (walking quadruped, Franka lift, imitation-learning HDF5 recording) with the actual ZED_X model mounted on the robot.

Franka lift task with ZED observations.

Using the sensor in your own script

The sl.sensor.camera package is importable without enabling the Kit extension (a plain sys.path insert is enough), and its helpers do the ZED-specific work:

1 import isaaclab.sim as sim_utils
2 from isaaclab.sensors import Camera
3 from sl.sensor.camera.isaaclab_utils import make_camera_cfg, unwrap_output, wait_for_camera_data
4 from sl.sensor.camera.utils import get_camera_paths, get_camera_usd_path
5 
6 # 1) Reference the ZED USD and place the WHOLE root (the cameras inherit the pose + baseline).
7 root = "/World/ZED_X"
8 zed = sim_utils.UsdFileCfg(usd_path=get_camera_usd_path("ZED_X"))
9 zed.func(root, zed)
10 
11 # 2) Attach Camera sensors to the authored stereo prims.
12 paths = get_camera_paths(root, "ZED_X")   # {"left": .../CameraLeft, "right": ..., "imu": ...}
13 left  = Camera(make_camera_cfg(paths["left"],  "ZED_X", "SVGA",
14                                data_types=["rgb", "distance_to_image_plane"]))
15 right = Camera(make_camera_cfg(paths["right"], "ZED_X", "SVGA", data_types=["rgb"]))
16 
17 # 3) Each step, read the tensors.
18 rgb_left = unwrap_output(left.data.output, "rgb")                       # (num_cameras, H, W, 3)
19 depth    = unwrap_output(left.data.output, "distance_to_image_plane")   # (num_cameras, H, W, 1), meters

When mounting the ZED on a robot, transform the USD root (or parent it under a robot link); never move the CameraLeft / CameraRight prims individually, or you break the stereo rig.

ZED Sim2Real camera model (experimental)

This feature is experimental and under active development; its behavior and calibration may change.

To make the RGB tensors look like a real ZED X camera (lens blur, vignetting, auto-exposure, color, sensor noise) instead of a clean render, pass --apply_zed_sim2real to the examples (optionally with --zed_sim2real_scene_lux <lux>). Each environment gets its own auto-exposure state, and the RGB tensors are degraded in place, which is useful for closing the sim-to-real gap in RL / imitation-learning pipelines. This is the same calibrated model the streaming path applies; it runs on the GPU via the sl_zed_sim2real library (a safe no-op if that library is not installed).

For the full argument list, output formats, and the flagship demo ports, see the examples README on GitHub.