STEREOLABS
Store

How to Make Mixed Reality VR Videos in Unity

Part 2: Installing the ZED Plugin

Before you start, you have to get your system ready to use your ZED. You can find a full installation guide here, but in short, make sure you have your Nvidia drivers up to date, plug in your camera, and run the ZED SDK installer.

The SDK installation ends by restarting your computer. Once you’re back in Windows, download the (ZED Unity plugin, open up your Unity project and import the plugin. To do this, go to Assets -> Import Package -> Custom Package.

How to import a package

Navigate to where you downloaded the ZED plugin. Import that file, and you’ll have a window open up to select what elements of the package you want to import. By default, they should all be selected. Hit “Import.” When you’re done, you should see the ZED folder listed in your Assets.

Unity's package importer

Note: One way to get a feel for the ZED is to import the package into a new Unity project and open the example scene in the ZED -> Examples -> Greenscreen -> Scenes folder. The prefab is already set up and there are several objects for playing around and testing occlusion. This tutorial focuses on adding a ZED to an existing project, and assumes that you either have an existing scene populated with other gameobjects, or that you’ll be building one in the same scene after finishing the tutorial. 

Now, with your target scene open, go into the ZED -> Examples -> GreenScreen -> Prefabs folder and drag the ZED_GreenScreen prefab into your scene.

Finding the ZED prefab

View of scene with prefab

Before we go further, the ZED_GreenScreen object has a component that’s worth exploring. Select ZED_GreenScreen, and in the Inspector, take a look at ZED Manager (ZEDManager.cs).

Image of ZED Manager script

The ZED Manager is responsible for setting up your ZED when you start your scene, and communicating with it at runtime. You don’t need to change these settings now, but keep them in mind for the future. To summarize:

  • Resolution: The input resolution from the camera, at 16:9. Note that the higher the resolution, the lower the FPS, even with a powerful computer; bigger images mean we can fit fewer of them through the USB 3.0 cable. The options are 480p (VGA) at 100FPS, 720p at 60FPS, 1080p at 30FPS, and 2k at 15FPS. Typically, 1080p30 is the best mix of resolution and frame rate for mixed reality.
  • Depth Mode: The quality of the depth map. Turning it up will make all of the effects more accurate, but will cost performance. ULTRA is recommended if your computer can handle it as it provides a substantial improvement over the next best level (QUALITY).
  • Enable Tracking: You can have the ZED track its own position on its own without attaching it to a tracked object. We’ll get into that more in the next section.
  • Enable Spatial Memory: If using the ZED’s tracking feature, this will remember features it sees as it moves around the room to help it reposition itself later.
  • Path Spatial Memory: Where you save the spatial memory file, if using it.
  • Everything else: The rest should remain unchanged for greenscreen-style mixed reality (unless you want to turn down the camera’s brightness).

Test the scene so far by hitting Play. In the Game window, you should see some combination of the real and virtual worlds. There is still much to do so it’s likely things will be out of alignment. But this will let you verify that your ZED is working and that the output is visible.

Troubleshooting

Some errors you may encounter at this stage, and their solutions:

  • If you are getting errors that the ZED plugin cannot find required files, it’s likely that either your Unity plugin version doesn’t match the ZED SDK plugin, or that something went wrong when you installed the ZED SDK or CUDA. To test the first, check the versions of the SDK and your plugin to ensure they match. To check the second, try running one of the apps included in the ZED SDK, like ZED Depth Viewer.
  • If you only see the view from an existing virtual camera, such as the one your VR headset sees, but the ZED’s blue light is on and you’ve given it time to initialize, check the Depth value of the existing virtual camera. That number needs to be lower than the ZED rig’s Depth value, which is 0 by default. Changing the virtual camera’s Depth to -1 should fix this.
    Another alternative is to create another Game window, set it’s Display value to “Display 2”, and change the ZED rig’s camera to output to Display 2. You can do this by expanding the ZED_GreenScreen object in the Hierarchy, selecting Camera_Left, and in the inspector, changing the Target Display

Next: Part 3: Positioning the Camera

Tags