Leap Motion’s hand tracking technology makes grabbing and poking virtual objects nearly as simple as in real life. Combine this with ZED Mini AR passthrough capabilities to bring your AR experience to the next level.
In this guide, we’ll add basic hand tracking and interactions to our Unity plugin’s Planetarium scene. This will let you knock planets around your room with your hands as though they were balloons.
We’ll assume you already have a ZED Mini attached to an Oculus Rift or HTC Vive. If not, follow this guide [link removed] for the Rift and this guide [link removed] for the Vive, skipping the 3rd test step if you wish.
The setup guide for your Leap Motion tells you to attach it to the center, where you’ve already placed your ZED Mini. Instead, attach it just below the ZED Mini. We’ll compensate for this later in Unity.
Your scene and hierarchy should look like this:
The Planetarium example scene already has the ZED_Rig_Stereo prefab in it, which handles the ZED’s image, tracking and headset output. By default, the Leap Motion rig also tracks the VR headset, using its own Unity camera. We have to switch it to follow the ZED rig instead.
Run the scene and put on your headset. You’ll see your real-life room, but with planets floating around it. Bring your hands in front of you and you’ll see Leap Motion’s hand tracking in action, drawing wires along your hands.
However, they won’t match your hands perfectly:
ZED Mini and Leap need an offset correction
That’s because the Leap Motion is mounted below the ZED, so their images don’t line up. Fixing this is easy, thanks to the Leap Offset object we made.
Now when you run the scene, the wires should already be lined up with your hands.
ZED Mini and Leap are now aligned in Unity
The default “hands” used in the Leap rig let you see where they are, but they can’t collide with anything in the scene. We want the opposite – invisible hands that cause collisions – so that it appears your real-life hands are colliding with the virtual objects.
The last step is to give something for the new physics-empowered hands to interact with. Adding colliders to the planets will do the trick.
Save, run the scene, and put on your headset. Reach out and poke a planet. Dribble it between your palms. Punch the sun to send it flying toward your buddy. Earth and the rest of the solar system is now in your hands!