Getting Started with UE

Building a mixed-reality experience with stereo pass-through and real-time depth interaction is complex and technically challenging. To ensure developers create stunning MR applications with UE4, we’re sharing an optimized Unreal Engine for AR/MR along with the ZED plugin for Unreal.

This custom modification of the Unreal Engine has been developed with the specific constraints of mixed reality in mind. It allows us to add several key technologies necessary to create believable and interactive AR/MR experiences:

  • Improved stereo pass-through: We’ve added technologies such as Video Async Reprojection (VAR) to eliminate rotational latency and improve visual comfort.
  • Interactive collisions: Virtual elements such as lasers or balls can collide with both static and dynamic objects in the real-world without having to scan the environment first.
  • Realistic shadows and lighting: Real-time depth and normal maps have been integrated in the rendering pipeline to offer a photorealistic integration of virtual content in the real-world. Virtual objects can cast shadows and project light on both static and dynamic real-world objects.
  • Depth occlusions: Occlusion of a virtual object by the real environment is required in order to trick human eyes into believing the virtual object is really present in the scene. By integrating long range depth sensing into UE4, virtual objects are naturally occluded by the surrounding environment.

Download the ZED SDK

The ZED SDK contains the core libraries that powers the ZED. It also contains tools to let you experiment its features and parameters. Download and install it from here.

Download the Engine

To gain access to the modified UE engine code, please follow the steps below:

  1. Create an account at GitHub: Sign up
  2. Create an account at Epic Games: Sign up
  3. Login to your Epic Games account and associate your GitHub account in the settings by entering your username into the GitHub account field.
  4. Login to your Github account and accept the Epic team invitation*. (Can take a few seconds to appear)
  5. Download or clone the modified UE4 engine code on Stereolabs GitHub. Be sure to pull/download the branch associated with your preferred Unreal Engine version (ex. 4.21-zed)

The link above will result in a 404 error if your github account is not linked to an Epic Game one. Make sure to follow all the instructions above.

*If the invitation isn’t sent, clicking on the following link seems to trigger it: Unreal Group. Note however that the Unreal Engine displayed on the page isn’t the one that you should download.

Build the Engine

Download the ZED Plugin for UE4

  • Download and install the latest ZED SDK.
  • Download or clone the ZED plugin source code. Be sure to pull/download the branch associated with the Unreal Engine version you downloaded, and the ZED SDK version you’re using (ex. UE4.21_ZedSdk2.8)
  • Create a Plugins folder in your project root directory and move the folder Stereolabs inside.
  • In your project root directory right click on the YourProjectName.uproject file and select Generate Visual Studio project files in the context menu.
  • Open Visual Studio and compile. Launch the project from the uproject file or Visual Studio.

Plugin Organization

The plugin contains the following modules:

  • Stereolabs: Implements an interface to the ZED SDK in UE4.
  • ZED: Contains the actor, player controller, pawn and utilities that allow to create and render mixed-reality video.
  • SpatialMapping: Contains a spatial mapping manager to use spatial mapping.
  • EnvironmentalLighting: Contains an environmental lighting manager to enable realistic lighting.
  • Devices: Contains actors to use VR controllers.

Explore Sample Projects

Explore the different sample projects available on GitHub and get started with MR development.