Getting Started with Unreal Engine and ZED

Introduction #

The ZED Plugin for Unreal allows developers to build AR/MR applications within Unreal Engine 5. It provides access to stereo video passthrough, depth sensing, positional tracking, spatial mapping and object detection features from the ZED SDK.

How to Get Started #

Prerequisites #

Do not hesitate to go on the Unreal Engine documentation page to know how to setup your Visual Studio Installation.

Note: The Object Detection and Body Tracking features are compatible with the ZED 2 and more recent cameras.

Installation #

  • Download or clone the ZED plugin for UE5.
  • In your project root directory right click on the ZEDSamples.uproject file and select Generate Visual Studio project files in the context menu.
  • Open Visual Studio and compile. Launch the project from the uproject file or Visual Studio.

If these options are not available, make sure your Visual Studio install is correctly setup.

How it works #

The plugin uses our C wrapper to interface with the SDK. This allows the usage of all the SDK features mentioned above, with either a video feed from a ZED Camera (via USB or streaming) or by reading a SVO file.

Available sample levels #

Explore the different sample levels provided in the project, like Spatial Mapping, Object Detection or Body Tracking.

  • L_BackgroundSubstraction : Background subtraction with plane occlusion or depth clamping.
  • L_BodyTrackingSingle : Body tracking of a single person in a virtual scene, with the Unreal Mannequin as an avatar.
  • L_BodyTrackingMulti : Body tracking of multiple persons in a virtual scene, with the Unreal Mannequin as an avatar.
  • L_CameraTracking : Camera tracking demo in a virtual scene.
  • L_Demo_Stage : Body tracking with legs inverse kinematics to improve the foot/ground interaction.
  • L_DynamicCrosshairOnMouse : Depth dynamic crosshair hovering on the environment, without spatial mapping.
  • L_NewHitTest : Depth-based hit test with ball stopping on the environment without spatial mapping.
  • L_ObjectDetection_2D : Object detection (human body model) with 2D frames.
  • L_ObjectDetection_3D : Object detection (human body model) with 3D boxes.
  • L_OpenNewLevel : Procedure to change active level, ZED-wise.
  • L_Passthrough : Simple AR passtrough vision showing a virtual object in a real scene, with depth occlusion.
  • L_PlaneDetection : Showcase of the SDK plane detection feature.
  • L_PointCloud : Point cloud mapping of the environment.
  • L_SpatialMapping : Visualization of the spatial mapping of the environment, and potentially creation of a static mesh out of the mapping.