How to Use YOLO with ZED

Introduction

This package lets you use YOLO (v3, v4, and more), the deep learning framework for object detection using the ZED stereo camera in Python 3 or C++.

The ZED and it’s SDK is now natively supported within the Darknet framework. It allows to use ZED 3D cameras with YOLO object detection, adding 3D localization and tracking to any Darknet compatible model.

Installation

ZED Yolo depends on the following libraries:

• ZED SDK and [Python API]
• Darknet (with cuDNN)
• OpenCV
• CUDA
• [Python 3]

ZED SDK

Install the ZED SDK and Python API.

cuDNN

Install cuDNN. Check the support matrix for the corresponding CUDA and driver version.

OpenCV

OpenCV binaries can be downloaded and install from opencv.org.

Alternatively, on Ubuntu :

sudo apt install pkg-config libopencv-dev

CMake

On Windows, download and install CMAKE using the binary available here.

On Ubuntu, cmake can be installed using the package manager, i.e : sudo apt install cmake

However the default version of cmake might be too old, it can easily be updated using the script (located in this repository):

sudo bash cmake_apt_update.sh

Compiling Darknet

Download and compile darknet, following the instructions:

• How to compile on Linux
• How to compile on Windows

ZED Support Using CMake (recommended)

If the ZED SDK is installed, CMake will automatically detect it and compile with the ZED support. During the CMake configuration, a message will confirm that the ZED SDK was found.

    ...
    -- A library with BLAS API found.
    -- ZED SDK enabled
    -- Found OpenMP_C: -fopenmp (found version "4.5")
    ...

ZED support Using Makefile

To enable the ZED support in YOLO using the Makefile, simply enable GPU and ZED_CAMERA, it’s also recommended to enable CUDNN for improved performances.

Launching YOLO 3D in C++

Download the yolo weights, yolov3 or yolov4 for instance, and put them in the local folder.

./uselib data/coco.names cfg/yolov4.cfg yolov4.weights zed_camera

SVO files are also supported :

./uselib data/coco.names cfg/yolov3.cfg yolov3.weights /path/to/svo/file.svo

How to use YOLO 3D in Python

The native support is currently only in C++.

For the Python version please refer to instructions in zed_python_sample

Setup ZED Python

Download and install the ZED Python wrapper following the instructions, to make sure everything works you sould try a sample.

Setup the application

The python script need to know the path to the darknet library, it can be modified here

  ...
  else:
      lib = CDLL("../libdarknet/libdarknet.so", RTLD_GLOBAL)
  lib.network_width.argtypes = [c_void_p]
  ...

• Download the model file, for instance Yolov3 tiny

wget https://pjreddie.com/media/files/yolov3-tiny.weights

Run the application

To launch the ZED with YOLO simply run the script :

python3 darknet_zed.py

The input parameters can be changed using the command line :

python3 darknet_zed.py -c <config> -w <weight> -m <meta> -t <threshold> -s <svo_file>

For instance :

python3 darknet_zed.py -c cfg/yolov3.cfg -w yolov3.weights -m cfg/coco.data -t 0.5

To display the help :

python3 darknet_zed.py -h

Using Docker

A DockerFile is provided in the Docker folder in the zed-yolo repository.