Green Screen VR Capture

Capturing VR in Mixed reality

In this tutorial, you will learn how to capture VR applications in mixed-reality using the ZED, an HTC Vive or Oculus Rift and a green screen.

Requirements

For this tutorial, you will need the following items:

  • ZED camera
  • Green screen
  • HTC Vive headset or an Oculus
  • 2x Vive controllers (or 1x Vive tracker and 2x controller) or 2x Oculus touch.

First, you will need to mount a controller/tracker on the ZED and make sure they are solidly attached. The controller will let you track the ZED in the same space as the HMD and allow you to move the camera while filming. To attach the ZED and controller/tracker, you can download and 3D print the support available here.

How It Works

To start recording mixed-reality footage, we will go through the following steps:

  • Create a new Unity project and import the ZED_Green_Screen prefab
  • Adjust ZED camera settings to improve image quality
  • Adjust chroma key to remove the green screen
  • Add the Vive headset

Note: If you experience framerate issues while using the Vive and ZED in Unity, please update SteamVR by opting in to the Beta Update.

Create a new Project

  • Open your Unity project, or create a new project.
  • In Edit > Project settings > Player, check Virtual Reality Supported.
  • In the Project panel, navigate to Asset > ZED > Prefabs.
  • Drag and drop the ZED_GreenScreen.prefab in the Hierarchy view.

  • In the Hierarchy view, select ZED_ GreenScreen > Camera_Left.

  • Click on the Play button. Your Unity interface should look like this.

Adjusting Camera Parameters

  • In the ZED Manager, make sure the camera is set in HD1080 mode (1920*1080 resolution).
  • By default, the camera exposure and white balance are adjusted automatically. Auto White Balance tend to add a green tint to the picture when placing the ZED in front of a green screen. Since you’re shooting in a controlled environment with fixed lighting, disable the Automatic mode for Exposure and White Balance in the Camera Control panel.

  • Depending on your lighting setup and preferences, adjust the camera settings to improve the picture and remove the green tint.

Adjusting Chroma Key Settings

In the Inspector panel, you can adjust your chroma key settings with the Green Screen Manager script.

Views

  • Select the FOREGROUND view, click on the color picker and select the green screen color in your image. Change the view to FINAL to see the result.

Color Range

  • Adjust the alpha matte with the Range and Smoothness settings. The easiest way to do is to set the sliders to 0, increase the Range and then the Smoothness.

  • Range defines the color interval that is accepted around the selected key, 0 being is the exact color.

  • Smoothness defines how color are blended around the selected range.

Refining the matte

  • Next, refine the edge of the matte with the Erosion, Black Clip, White Clip, Edge Softness and Despill settings. Use the ALPHA view to make these adjustments.

  • The Erosion setting erodes the edges of the matte to remove the remaining green areas off the matte.

    Without erosion With erosion
  • White Clip and Black Clip: Alpha values above the White Clip value will be set to 1 (fully opaque), and alpha values below the Black Clip value will be set to 0 (fully transparent).

    Without white clip With white clip
  • Edge Softness feathers the edge of the matte to avoid aliased edges and improve the integration of the key in the background.

    Without edge softness With edge softness
  • Despill reduces the green screen color reflections on the key. Change the view to FINAL before adjusting color despill.

    Without despill With despill
  • Finally, store your chroma key parameters by clicking on Save.

Garbage Matte

The Garbage matte script lets you extend the virtual world beyond the green screen limited size. It allows you to define a region in space and exclude image pixels that do not belong to this region. To use camera tracking with the garbage matte, you need to activate tracking before setting your garbage matte.

  • Add the script ZEDGarbageMatte on ZED_GreenScreen > Camera_Left.

  • Define your green screen region by clicking on the image to place your markers.

  • Repeat the previous step until the whole green screen area is covered with a white mesh.

  • Click on Apply Garbage Matte to exclude the outside region of the mesh (set to transparent).

  • To save and load markers, use the Save/Load option.

Note: If the area is not covered correctly, use Remove Last Point to disable the garbage matte and remove the last marker. You can also adjust the position of a specific marker with a right click to select a sphere and a left click on a new position.

Adding the Vive headset

Interfacing the ZED with the Vive

  • Download and import the SteamVR Unity package.
  • Add the [CameraRig] prefab located in the folder SteamVR / Prefabs. If you are using Unity 5.6 with SteamVR 1.2.1, you need to add the script SteamVR_UpdatePoses to the GameObject [CameraRig] -> Camera (head) -> Camera (eye).
  • Right click in the Hierarchy view, and then on Create Empty.
  • Drag the ZED_GreenScreen prefab and drop it on the GameObject.

  • Launch the sample scene with the Play button.

  • The GameObject ZED_GreenScreen contains the script ZEDSteamVRPadManager. It loads the calibration configuration created by the calibration software (beta) and delays the pads to be synchronized with the images of the ZED.

  • To set up your calibration manually, disable the script ZEDSteamVRPadManager and add a SteamVR_Tracked_Object to the parent. It this script parameters, you can select the controller attached to the camera in the Index list. Finally, manually adjust the transform of the ZEDGreenScreen to get a matching physical and virtual world.

Note: Oculus Rift is also supported through SteamVR plugin.

Aligning the real and virtual world

  • Finally, let’s align the real world captured by the camera with the virtual world displayed in Unity. This is called the “calibration” step.
  • Put a second Vive controller in front of the ZED. If the system is not calibrated, you should see that the real and virtual controllers do not overlap.
  • Click on ZED_GreenScreen and adjust the Transform values till the position and rotation of the virtual and real controller match. Try to put the controller closer to the camera and adjust translation values, then move further away and adjust rotation.
  • You may need to repeat the previous steps several times to achieve a good calibration of the real and virtual world.
  • To save your configuration, click on Save Camera Offset in the Inspector view > ZED Position Manager. This will save a ZED_Position_Offset.conf file.

In the latest SDK update, we provide a semi-automated calibration app that lets you create the offset file automatically. You can download a beta version here.