Getting Started with Unreal Engine and ZED

Introduction

The ZED Plugin for Unreal allows developers to build AR/MR applications within Unreal Engine 5. It provides access to stereo video passthrough, depth sensing, positional tracking, spatial mapping and object detection features from the ZED SDK.

Compatibility

ZED Cameras

The Object Detection and Body Tracking features will only work with ZED2 and more recent cameras.

Version 1.2

  • Unreal Engine 5.1
  • ZED SDK 4.0
  • New Body models with hands tracking and better rotations for ankles and wrists
Older versions for ZED SDK v3.8

The 1.2 version is the most recent version, made to work with the ZED SDK 4.0. To use older versions and for more information about the SDK 3.8, please visit the previous documentation

How it works

The plugin uses our C language wrapper to interface with the SDK. This allows the usage of all the SDK features mentioned above, with either a video feed from a ZED Camera (via USB or streaming) or by reading a SVO file.

Available samples

  • L_BackgroundSubstraction : Background subtraction with plane occlusion or depth clamping.
  • L_BodyTracking : Body tracking in a virtual scene, with the Unreal Mannequin as an avatar.
  • L_CameraTracking : Camera tracking demo in a virtual scene.
  • L_Demo_Stage : Body tracking with legs inverse kinematics to improve the foot/ground interaction.
  • L_DynamicCrosshairOnMouse : Depth dynamic crosshair hovering on the environment, without spatial mapping.
  • L_NewHitTest : Depth-based hit test with ball stopping on the environment without spatial mapping.
  • L_ObjectDetection_2D : Object detection (human body model) with 2D frames.
  • L_ObjectDetection_3D : Object detection (human body model) with 3D boxes.
  • L_OpenNewLevel : Procedure to change active level, ZED-wise.
  • L_Passthrough : Simple AR passtrough vision showing a virtual object in a real scene, with depth occlusion.
  • L_PlaneDetection : Showcase of the SDK plane detection feature.
  • L_PointCloud : Point cloud mapping of the environment.
  • L_SkeletonAvatar : Visualization of the skeleton tracking data from the SDK.
  • L_SkeletonTrackinSpatialMapping : Usage of both the skeleton tracking and the spatial mapping features.
  • L_SpatialMapping : Visualization of the spatial mapping of the environment, and potentially creation of a static mesh out of the mapping.