Green Screen VR Capture

Capturing VR in Mixed reality

In this tutorial, you will learn how to capture VR applications in mixed-reality using the ZED, an HTC Vive and a green screen.

How it works

For this tutorial, you will need the following items:

  • ZED camera
  • Green screen
  • HTC Vive headset
  • 2x Vive controllers (or 1x Vive tracker and 1x controller)

First, you will need to mount a Vive controller/tracker on the ZED and make sure they are solidly attached. The controller will let you track the ZED in the same space as the HMD and allow you to move the camera while filming.

To start recording mixed-reality footage, we will go through the following steps: - Create a new Unity project and import the ZED Green Screen prefab - Adjust the ZED camera settings to improve image quality - Adjust the chroma key

Create a new Project

  • Open your Unity project, or create a new project.
  • In Edit > Project settings > Player, check Virtual Reality Supported.
  • In the Project panel, navigate to Asset > ZED > Prefabs.
  • Drag and drop the ZED_GreenScreen.prefab in the Hierarchy view.

  • In the Hierarchy view, select ZED_ GreenScreen > Camera_Left.

  • Click on the Play button. Your Unity interface should look like this.

Adjusting Camera Parameters

  • In the ZED Manager, make sure the camera is set in HD1080 mode (1920*1080 resolution).
  • By default, the camera exposure and white balance are adjusted automatically. Auto White Balance tend to add a green tint to the picture when placing the ZED in front of a green screen. Since you’re shooting in a controlled environment with fixed lighting, disable the Automatic mode for Exposure and White Balance in the Camera Control panel.

  • Depending on your lighting setup and preferences, adjust the camera settings to improve the picture and remove the green tint.

Adjusting Chroma Key Settings

In the Inspector panel, you can adjust your chroma key settings with the Green Screen Manager script.

Views

  • Select the FOREGROUND view, click on the color picker and select the green screen color in your image. Change the view to FINAL to see the result.

Color Range

  • Adjust the alpha matte with the Range and Smoothness settings. The easiest way to do is to set the sliders to 0, increase the Range and then the Smoothness.

  • Range defines the color interval that is accepted around the selected key, 0 being is the exact color.

  • Smoothness defines how color are blended around the selected range.

Refining the matte

  • Next, refine the edge of the matte with the Erosion, Black Clip, White Clip, Edge Softness and Despill settings. Use the ALPHA view to make these adjustments.

  • The Erosion setting erodes the edges of the matte to remove the remaining green areas off the matte.

    Without erosion With erosion
  • White Clip and Black Clip: Alpha values above the White Clip value will be set to 1 (fully opaque), and alpha values below the Black Clip value will be set to 0 (fully transparent).

    Without white clip With white clip
  • Edge Softness feathers the edge of the matte to avoid aliased edges and improve the integration of the key in the background.

    Without edge softness With edge softness
  • Despill reduces the green screen color reflections on the key. Change the view to FINAL before adjusting color despill.

    Without despill With despill
  • Finally, store your chroma key parameters by clicking on Save.

Adding the Vive headset

Interfacing the ZED with the Vive

  • Download and import the SteamVR Unity package.
  • In the Hierarchy panel, delete the Main Camera.
  • Add the [CameraRig] prefab located in the folder SteamVR / Prefabs. If you are using Unity 5.6 with SteamVR 1.2.1, you need to add the script SteamVR_UpdatePoses to the GameObject [CameraRig] -> Camera (head) -> Camera (eye).
  • Right click in the Hierarchy view, and then on Create Empty.
  • Drag the ZED_GreenScreen prefab and drop it on the GameObject.

  • Click on the GameObject, and add a SteamVR_TrackedObject component.

  • Launch the sample scene with the Play button

  • In the Inspector panel, select the index of the controller/tracker you mounted on the camera.

Aligning the real and virtual world

  • Finally, you need to align the real world captured by the camera and the virtual world displayed in Unity. This is the “calibration” step.
  • Put a second Vive controller in front of the ZED.
  • Click on ZED_GreenScreen and adjust the Transform values till the position and rotation of the virtual and real controller match. Try to put the controller close to the camera and adjust translation values, then move further away and adjust rotation.
  • You may need to repeat the previous steps to achieve a good calibration of the real and virtual world.
  • To save your configuration, click on Save Camera Offset in the Inspector view > ZED Position Manager.