Green Screen VR Capture

Capturing VR in Mixed Reality

In this tutorial, you will learn how to capture VR applications in mixed-reality using the ZED, an HTC Vive or Oculus Rift and a green screen. For an in-depth article on capuring VR content with an external camera, read this tutorial.

The main advantage of using depth data from the ZED camera is pixel occlusion and lighting. This means that the depth of every pixel in the camera’s view will be compared against its corresponding virtual pixel, so the nearest one will be rendered. In addition, you can enhance your real scene with virtual lights and shadows.

Requirements

For this tutorial, you will need the following items:

  • ZED camera
  • Green screen
  • HTC Vive headset or Oculus Rift
  • 2x Vive controllers, 1x Vive Tracker and 2x controllers, or 2x Oculus Touch controllers.

First, you will need to mount a controller/tracker onto the ZED and make sure they are firmly attached. The controller will let you track the ZED in the same space as the HMD and allow you to move the camera while filming. To attach the ZED and controller/tracker from Oculus or Vive, you can 3D print the following mount (download STL, OBJ).

If you don’t have access to a 3D printer, you can attach the devices with clamps, tape, etc. But you must ensure two things: First, that your attachment mechanism doesn’t occlude too many of the controller/tracker’s sensors. Second, your attachment must be firm (no wobbling) or else you will have to recalibrate constantly.

How It Works

To start recording mixed reality footage, we will go through the following steps:

  • Create a new Unity project and import the ZED_Green_Screen prefab.
  • Adjust ZED camera settings to improve image quality.
  • Adjust chroma key to remove the green screen.
  • Add a VR headset.

Note: If you experience random freeze while using Vive with ZED, update your NVIDIA drivers (version 390.77 or greater for GeForce cards) to fix the issue. Read more.

Create a new Project

  • Open your Unity project, or create a new project.
  • Follow the instructions on importing the ZED Unity package.
  • In Edit > Project settings > Player, check Virtual Reality Supported.
  • In the Project panel, navigate to Asset > ZED > Examples > GreenScreen > Prefabs. Drag and drop the ZED_GreenScreen.prefab in the Hierarchy view.
  • In the Hierarchy view, select ZED_ GreenScreen > Camera_Left.

  • Click on the Play button. Your Unity interface should look like this:

Adjusting Camera Parameters

  • In the ZED_ GreenScreen > ZED Manager (Script), make sure the camera is set in HD1080 mode (1920*1080 resolution). Changing the resolution requires to restart the scene.
  • After selecting ZED_ GreenScreen > Camera_Left, click on the Camera Control button of the Green Screen Manager in the Inspector view.
  • By default, the camera exposure and white balance are adjusted automatically. Auto White Balance tends to add a green tint to the picture when placing the ZED in front of a green screen. Since you’re shooting in a controlled environment with fixed lighting, you can disable the Automatic settings and manually adjust the parameters of the Camera Control panel.

  • Hit Save once you’re happy with your settings, so they can be easily loaded later after restarting Unity.

Adjusting Chroma Key Settings

In the Inspector panel, you can adjust your chroma key settings with the Green Screen Manager script. Below are explanations of what each setting does and when to adjust it.

Views

Select the FOREGROUND view to get a better feel for what remains visible in your scene and what doesn’t. This can be useful for setting the critical settings in the next section. Change the view to FINAL to see the result.

Color Range

The Color, Range and Smoothness values are the most important and where you’ll likely spend the most time adjusting. When the script goes through each frame, it will check each pixel’s color to see if it should turn it invisible.

  • If you’re using a background that’s not green, adjust the Color setting to match. Otherwise, adjust the shade of green to better fit the screen’s exact color and your lighting setup.

  • Range indicates how close a color on the spectrum can be to the exact color of your background. You need to add some range because even the highest quality green screen studio has shading to it.

  • Smoothness defines how colors just outside the range get blended in. With zero smoothness, a color just outside the range is completely visible. With more smoothness, it will be partially visible. You’ll get further control over how smoothness effects the image in the next section.

The easiest way to find a good Range and Smoothness is to start the scene, set both values to 0, adjust the Range, then adjust the Smoothness.

Refining the Matte

  • Next, refine the edge of the matte with the Erosion, Black Clip, White Clip, Edge Softness and Despill settings. The ALPHA view can be helpful for making these adjustments.

  • The Erosion setting erodes the edges of the matte to remove the remaining green areas.

    Without erosion With erosion
  • White Clip and Black Clip: Alpha values above the White Clip value will be set to 1 (fully opaque), and alpha values below the Black Clip value will be set to 0 (fully transparent). In other words, White Clip makes nearly opaque pixels fully opaque, and Black Clip does the same for nearly invisible pixels. This helps you remove noise that results from your smoothness setting.

    Without white clip With white clip
  • Despill reduces the green screen color reflections on the key. Increase this if you have a slight green hue showing on your subject, but be careful not to use more than you need or the other colors will appear less vibrant. Change the view to FINAL before adjusting color despill.

    Without despill With despill
  • Finally, store your chroma key parameters by clicking on Save.

Garbage Matte

The Garbage matte script lets you extend the virtual world beyond the green screen’s limited size. It allows you to define a region in space and exclude image pixels that do not belong to this region. To use camera tracking with the garbage matte (as we don’t do in this demo as we’re using the headset) you need to activate tracking before setting your garbage matte. Tracking can be activated by clicking on the ZED_Greenscreen prefab, and enabling it in the ZED Manager compponent.

  • Click on Enable Garbage Matte

  • Start your scene.

  • Make sure Place Markers is selected.

  • Define your green screen region by clicking on the image in your Game display window to place your markers. You should see spheres appear at each place you click.

  • Repeat the previous step until the whole green screen area is covered with a white mesh.

  • Click on Apply Garbage Matte to exclude the outside region of the mesh (set to transparent).

  • To save and load markers, use the Save/Load option.

Note: If the area is not covered correctly, use Remove Last Marker to disable the garbage matte and remove the last marker. You can also adjust the position of a specific marker with a right click to select a sphere and a left click on a new position.

Adding the VR headset

Interfacing the ZED with SteamVR

SteamVR is required for the HTC Vive and optional for the Oculus Rift.

  • Download and import the SteamVR Unity package.
  • Add the [CameraRig] prefab located in the folder SteamVR / Prefabs.
  • Right click in the Hierarchy view, and then on Create Empty to create a GameObject.
  • Drag the ZED_GreenScreen prefab and drop it on the GameObject. Then make sure its Position and Rotation values are all zero in the Inspector panel.

  • Launch the sample scene with the Play button.

  • The GameObject ZED_GreenScreen contains the script ZEDSteamVRControllerManager. It loads the calibration configuration created by the calibration software if you choose to use it (beta). It also adds a controllable delay to the movement of the virtual controllers, which is important as the difference in latency between the controllers and ZED’s output cause the real and virtual controllers to fall out of alignment if moved quickly.

  • ZEDSteamVRControllerManager adds a delay by making a copy of the target controller that lags behind the real one. To achieve the desired effect, disable the SteamVR_RenderModel or MeshRenderer component of each controller it is copying.

Interfacing the ZED with Oculus Rift

The Oculus Integration plugin, which we’ll be using, also supports green screen mixed reality with the ZED natively. It lacks per-pixel occlusion features but supports other features such as using a third Oculus controller. Read more.

  • Download and import the Oculus Integration Unity Package.
  • Add the OVRCameraRig prefab to your Hierarchy, located in OVR/Prefabs. Expand it to reveal its children.
  • Drag the ZED_GreenScreen prefab onto either LeftHandAnchor or RightHandAnchor depending on which controller your ZED is affixed to.

  • Remove the ZED SteamVR Controller Manager component.

  • If your controllers have visible models attached to them, replace the component you just removed with ZED Oculus Controller Manager. Click and drag the objects with the MeshFilter/MeshRenderer displaying the object into the fields for Model Left Controller and Model Right Controller. You can use this to add a delay to the visible controllers which is important as the difference in latency between the controllers and ZED’s output cause the real and virtual controllers to fall out of alignment if moved quickly.

  • ZEDOculusControllerManager adds a delay by making a copy of the target controller that lags behind the real one. To achieve the desired effect, disable the the MeshRenderer or SkinnedMeshRenderer component component of each controller it is copying.

Manual Calibration

We provide a semi-automated calibration app for SteamVR that lets you create an offset file automatically and calibrate the relative position of the ZED and Vive/Oculus play space. You can download a beta version here.

If you prefer to adjust the offset manually or you are using Oculus Rift, follow the steps below:

  • To set up your calibration manually, disable/remove the script ZEDSteamVRControllerManager and add a SteamVR_Tracked_Object to the parent.

    In these script parameters, you can select the controller attached to the camera in the Index list. If you have created a calibration file with our beta app before, also disable the ZEDOffsetController component in ZED_GreenScreen.

  • Run the scene.

  • Put a second controller in front of the ZED. Since the system is not calibrated, you should see that the real and virtual controllers do not overlap. (If the calibration is way off, the controller may be completely out of view.)

  • Click on ZED_GreenScreen and adjust the Transform values until the position and rotation of the virtual and real controller match. Try to put the controller closer to the camera and adjust translation values, then move further away and adjust rotation.

  • You may need to repeat the previous steps several times to achieve a good calibration of the real and virtual world. Also try moving the controller to both edges of the screen.

  • To save your configuration, click on Save Camera Offset in the Inspector view > ZED Offset Controller. This will save a ZED_Position_Offset.conf file.