Tutorial - Using 3D Object Detection

This tutorial shows how to use the ZED SDK to detect, classify and locate objects in 3D (compatible with ZED 2 only). It works with both a static or moving camera.

Introduction

In this tutorial, we will use the Object Detection module of the ZED SDK. This AI module uses the GPU to perform deep neural network computations. On platforms with limited amount of memory such as Jetson Nano, it is advised to disable the GUI to improve performance and avoid memory bottlenecks.

Getting Started

  • First, download the latest version of the ZED SDK.
  • Download the C++ or Python sample code.
  • Follow the instructions on how to build your project in C++ or Python on Windows and Linux.

Code Overview

Create a camera

As in previous tutorials, we create, configure and open the ZED 2. Please note that the ZED 1 is not compatible with the object detection module.

// Create ZED objects
Camera zed;
InitParameters initParameters;
initParameters.camera_resolution = RESOLUTION::HD720;
initParameters.depth_mode = DEPTH_MODE::PERFORMANCE;
initParameters.sdk_verbose = true;

// Open the camera
ERROR_CODE zed_error = zed.open(initParameters);
if (zed_error != ERROR_CODE::SUCCESS) {
	std::cout << "Error " << zed_error << ", exit program.\n";
	return 1; // Quit if an error occurred
}

Enable Object detection

We specify here the Object Detection parameters. Note that enabling object tracking needs camera positional tracking to be active to be able to track objects movements independently from camera motion.

// Define the Objects detection module parameters
ObjectDetectionParameters detection_parameters;
detection_parameters.image_sync = true;
detection_parameters.enable_tracking = true;
detection_parameters.enable_mask_output = true;

// Object tracking requires camera tracking to be active
if (detection_parameters.enable_tracking)
	zed.enablePositionalTracking();

Then we start object detection which will launch an AI model. This operation can take a few seconds. The first time the module is used, the model will be optimized for your hardware and this can take up to a few minutes. The model optimization operation is done only once.

std::cout << "Object Detection: Loading Module..." << std::endl;
zed_error = zed.enableObjectDetection(detection_parameters);
if (zed_error != ERROR_CODE::SUCCESS) {
	std::cout << "Error " << zed_error << ", exit program.\n";
	zed.close();
	return 1;
}

Object detection is now active.

Retrieve object data

The object confidence threshold can be adjusted at runtime to select only the relevant objects depending on your application. Since the parameters have been set to image_sync, for each grab call, the image will be fed into the AI module that will output the detected objects for each frame.

// Set runtime parameter confidence to 40
ObjectDetectionRuntimeParameters detection_parameters_runtime;
detection_parameters_runtime.detection_confidence_threshold = 40;

Objects objects;

// Grab new frames and detect objects
while (zed.grab() == ERROR_CODE::SUCCESS) {
	zed_error = zed.retrieveObjects(objects, detection_parameters_rt);

	if (objects.is_new) {
        // Count the number of detected objects
        std::cout << objects.object_list.size() << " Object(s) detected"

        // Display 3D location of one object    
        std::cout << " 3D position: " << first_object.position;

        // Display 3D bounding box coordinates        
        std::cout << " Bounding Box 3D \n";
        for (auto it : first_object.bounding_box)
          std::cout << "    " << it;    
	}
}

Disable modules and exit

Once the program is over, the modules can be disabled and the camera closed. This step is optional since the zed.close() will take care of disabling all the modules. The close() function is also called automatically by the destructor if necessary.

// Disable object detection and close the camera
zed.disableObjectDetection();
zed.close();
return 0;

And this is it!

You can now detect objects in 3D with your ZED 2.