How to Make Mixed Reality VR Videos in Unity

Part 6: Background Subtraction

Now to configure the mixed reality green screen. For each pixel in the video feed, the package will ignore any of “real” pixels that are close enough to the color of your choosing. To set this up, click on Camera_Left, which is parented to the ZED_GreenScreen prefab. You’ll see a number of options under the “Green Screen Manager” script in the Inspector.

Tip: Proper lighting is essential to a quality Chroma key/green screen mixed reality setup. Lighting is a complicated art that’s outside the scope of this tutorial, but keep this in mind if you’re having trouble getting the background to subtract properly – Unity may not be where to tweak things.

Note: Because you have access to depth, there may technically be situations where a green screen isn’t needed. If the entirety of your VR environment is smaller than your real environment, with no parts of the real background that would be closer than the virtual background, then the entire background will be occluded by the real one. However, a green screen is still preferable in those situations to remove depth errors, so this option should only be explored if filming with a green screen is a big obstacle for you.

Most important are Color, Range and Smoothness. When the script goes through each frame, it’ll check each pixel’s color to see if it should turn it invisible. If you’re using a background that’s not green, adjust the Color setting to match. You can also try tweaking the shade of green to better fit the screen’s exact color and your lighting setup.

Run your scene and change each of the values below. However, as you do so, remember that stopping the scene will cause Unity to revert the settings to the way they were before you started. So either write down your final settings, or use the Save button to save them and Load them after you’ve stopped your scene.

As you change the values, you can also change the View setting to help you visualize the changes you’re making. The Alpha view is especially useful for settings that deal with which pixels are removed.

Range indicates how close a color on the spectrum can be to the exact color you chose to be invisible. You need some range because even the highest quality green screen studio has shading to it. But set it too high and parts of your subject will start to turn invisible, too.

Smoothness makes colors just outside the range get blended in. With zero smoothness, a color just outside the range is completely visible. With more smoothness, it’ll be partially visible. This is another setting to play with at runtime.

Tip: If you’re not sure where to start, set both the Range and Smoothness to zero. Then raise the Range until it’s as good as you can make it look, then raise the Smoothness until it’s near perfect.

The rest of the settings are less critical, but help you solve specific issues:

Erosion: This setting removes the edges of real objects in the video stream where they border with the green area. This is useful because those edges will sometimes be blended together, resulting in your subjects having thin green outlines. Play with this setting if that’s the case.

White Clip/Black Clip: Alpha values (transparency) above the White Clip value will be set to 1 (fully opaque), and alpha values below the Black Clip value will be set to 0 (fully transparent). In other words, White Clip makes nearly opaque pixels fully opaque, and Black Clip does the same for nearly invisible pixels. This helps you remove noise that results from your smoothness setting.

Despill: This makes the non-green pixels look even less green (or whichever color you’re using). This is useful because you often have slight green reflections on a subject, even on clothes and other non-reflective materials, thanks to the bright lights and the big green walls. However, raising this setting makes things look less vibrant, so don’t raise it any more than you need.

Garbage Matte: The Garbage matte lets you extend the virtual world beyond the green screen’s limited size. In short, it lets you paint a box where you’ll see your subject, and everything outside that box will be automatically deleted as if it were a green screen. This is useful if your green screen is too small to fill everything the ZED can see.

When you enable the Garbage Matte setting, you’ll see a few more buttons appear.

While the game is running, press Place Markers, and click on the four corners of your Game window (not your Scene window) where you want this “box” to appear. Each time you click, you should see a ball appear there. (If not, it may be behind a real object.) Once you have four of them, it should create a white plane that represents the box.

Click Apply and it’ll start working. To preserve this matte for later, you’ll have to hit Save to save all your settings, and load them after you stop the scene.

Next: Part 7: Virtual Lighting