Capturing Depth image from Stereo Vision using Matlab

Introduction

The purpose of this post is to explain how to use the ZED stereo camera and its SDK to capture depth maps with Matlab. We use the ability of Matlab to execute C++ code through an intermediate file called MEXA file.

You will find the complete source code on Stereolabs’ Github page.

If you simply want to use the ZED stereo camera with Matlab (without the ZED SDK) you can refer to this article.

Prerequisites

You must have Matlab installed on your system and of course, the ZED SDK needs to be installed too.

Code

When downloading the source code from GitHub, you will find two subfolders. The first one, ‘matlab’, contains a file named ‘ZED_Camera.m’, a standard Matlab file wich will call the ZED functions. The second subfolder, ‘src’, contains the files to build the application.
In the ‘mex’ subfolder, there is a file named ‘mexZED.cpp’ which contains the ZED SDK functions and interoperability with the Matlab format. This file is referred to as a MEX file. It re-implements the majority of the ZED SDK functions. This file is provided as a basic container and can be complemented with your own functions. Since Matlab does not handle image representation the same way the ZED SDK does, we need additional formatting functions.In the file ‘ZED_Camera.m’ the ZED functions are reachable through mexZED(‘..’). Check the ‘mexZED.cpp’ to see the list of functions available.

Build procedure

You now want to compile the mexZED.cpp file in a MEXA file format which can be used by Matlab. Usually, MEX files are compiled directly by Matlab but since we need to link several libraries (ZED, OpenCV) it is better to use CMake.
On Linux

Open a terminal in zed-Matlab directory and execute the following command:

export MATLAB_ROOT=/usr/local/MATLAB/R2012b
mkdir build
cd build
cmake ../src
make
make install
On Windows

In cmake GUI, set ‘where is the source code’ as ‘zed-matlab\src’ and ‘Where to build the binairies ‘ as ‘zed-matlab\build’.

Then be sure to specify a x64 version of Visual Studio, the ZED SDK is only compatible with x64 compiler. Click ‘Configure’, you may encounter an error saying ‘MATLAB not found…nothing will be built.’, don’t worry, you juste need to specify the variable ‘MATLAB_ROOT’ (ex : MATLAB_ROOT=C:/Program Files/MATLAB/R2014b).

Click, ‘Generate’, then go to the folder build and open ‘MEX.sln’ with visual studio. Set the Solution Configuration to Release and build mexZED, then build INSTALL.

Run the sample

You can now find the generated MEXA file in the matlab directory (e.g. ‘\zed-matlab\matlab’), you can launch Matlab and run the ZED_Camera.m file.

zedmatlab_2

 

  • Geoffrey Flynn

    When I compile the MEX.sln; I am getting a compile error “C2678 binary ‘==’ ; no operator found which takes a left-hand operator of type ‘pix_iterator_2d

    it seems to be related to the line in mexZED.cpp:

    pix_iterator_2d it_src1(static_cast (pBeg), size.width, size.height, pitch);
    pix_iterator_2d it_src2(static_cast (pBeg), size.width, size.height, pitch);
    it_src2.end();
    pix_iterator_2d it_dest(static_cast (pBegOut), size.width, size.height);

    std::copy(it_src1, it_src2, it_dest);

    Any help on this issue would be great.

    • HI Geoffrey,
      Which version of visual studio do you use ? we test it on VS2013 and it works well.

      • José Pablo Sanchez

        I got the same problem. I am also using VS2014, opencv 2.4.12 the Cmake process succeded but when I try to build the Mex.sln I get the same error code.

        • For the moment the ZED SDK is only compatible with VS2013.
          We use CUDA 7.0 which is not supported by the newest version of VS.

          • José Pablo Sanchez

            I just made sure I’m using VS2013 and CUDA 7.0. I’m still having the same error:
            Error 12 error C2678: binary ‘==’ : no operator found which takes a left-hand operand of type ‘pix_iterator_2d’ (or there is no acceptable conversion) C:Program Files (x86)Microsoft Visual Studio 12.0VCincludexutility 2125 1 mexZED Any suggestions?

          • It seems that the error occurs only on Debug mode, I advise you to switch to the Release mode. Meanwhile we are looking for a proper solution.

          • We fixed the bug. Please pull the changes from Github.
            Let us know if you encounter other problems.

    • José Pablo Sanchez

      Hey Geoffrey, did you solve this problem already?
      Regards!

  • Kęstas Rimkus

    Hi, all.
    I wann to compile the mex.sln by this tutoria, but i got some LNK2019 errors. I’m using VS2013 and CUDA 7.0.

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxGetData referenced in function “struct mxArray_tag * __cdecl floatmat_to_mat(class cv::Mat &)” (??$floatmat_to_mat@$04@@YAPAUmxArray_tag@@AAVMat@cv@@@Z)

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxIsChar referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxGetPr referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxSetField_700 referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxCreateNumericMatrix_700 referenced in function “struct mxArray_tag * __cdecl floatmat_to_mat(class cv::Mat &)” (??$floatmat_to_mat@$04@@YAPAUmxArray_tag@@AAVMat@cv@@@Z)

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxCreateNumericArray_700 referenced in function “struct mxArray_tag * __cdecl rgbimage_to_mat(class cv::Mat &)” (??$rgbimage_to_mat@$07@@YAPAUmxArray_tag@@AAVMat@cv@@@Z)

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxCreateDoubleMatrix_700 referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxCreateDoubleScalar referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxGetString_700 referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxCreateString referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol _mxCreateStructMatrix_700 referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol _mexErrMsgTxt referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol _mexPrintf referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: int __thiscall sl::zed::Mat::getDataSize(void)const ” (?getDataSize@Mat@zed@sl@@QBEHXZ) referenced in function “class cv::Mat __cdecl sl::zed::slMat2cvMat(class sl::zed::Mat)” (?slMat2cvMat@zed@sl@@YA?AVMat@cv@@V312@@Z)

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: __thiscall sl::zed::Camera::Camera(enum sl::zed::ZEDResolution_mode,float,int)” (??0Camera@zed@sl@@QAE@W4ZEDResolution_mode@12@MH@Z) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: __thiscall sl::zed::Camera::Camera(class std::basic_string<char,struct std::char_traits,class std::allocator >)” (??0Camera@zed@sl@@QAE@V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: __thiscall sl::zed::Camera::~Camera(void)” (??1Camera@zed@sl@@QAE@XZ) referenced in function “public: void * __thiscall sl::zed::Camera::`scalar deleting destructor'(unsigned int)” (??_GCamera@zed@sl@@QAEPAXI@Z)

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: enum sl::zed::ERRCODE __thiscall sl::zed::Camera::init(enum sl::zed::MODE,int,bool,bool,bool)” (?init@Camera@zed@sl@@QAE?AW4ERRCODE@23@W4MODE@23@H_N11@Z) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: bool __thiscall sl::zed::Camera::grab(enum sl::zed::SENSING_MODE,bool,bool)” (?grab@Camera@zed@sl@@QAE_NW4SENSING_MODE@23@_N1@Z) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: class sl::zed::Mat __thiscall sl::zed::Camera::retrieveImage(enum sl::zed::SIDE)” (?retrieveImage@Camera@zed@sl@@QAE?AVMat@23@W4SIDE@23@@Z) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: class sl::zed::Mat __thiscall sl::zed::Camera::retrieveMeasure(enum sl::zed::MEASURE)” (?retrieveMeasure@Camera@zed@sl@@QAE?AVMat@23@W4MEASURE@23@@Z) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: class sl::zed::Mat __thiscall sl::zed::Camera::normalizeMeasure(enum sl::zed::MEASURE,float,float)” (?normalizeMeasure@Camera@zed@sl@@QAE?AVMat@23@W4MEASURE@23@MM@Z) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: struct sl::zed::StereoParameters * __thiscall sl::zed::Camera::getParameters(void)” (?getParameters@Camera@zed@sl@@QAEPAUStereoParameters@23@XZ) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: void __thiscall sl::zed::Camera::setConfidenceThreshold(int)” (?setConfidenceThreshold@Camera@zed@sl@@QAEXH@Z) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: struct sl::zed::resolution __thiscall sl::zed::Camera::getImageSize(void)” (?getImageSize@Camera@zed@sl@@QAE?AUresolution@23@XZ) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: bool __thiscall sl::zed::Camera::setSVOPosition(int)” (?setSVOPosition@Camera@zed@sl@@QAE_NH@Z) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: int __thiscall sl::zed::Camera::getSVONumberOfFrames(void)” (?getSVONumberOfFrames@Camera@zed@sl@@QAEHXZ) referenced in function _mexFunction

    1>mexZED.obj : error LNK2019: unresolved external symbol “public: void __thiscall sl::zed::Camera::setDepthClampValue(int)” (?setDepthClampValue@Camera@zed@sl@@QAEXH@Z) referenced in function _mexFunction

    Any suggestions?

    • Kęstas Rimkus

      Found problem. I’, using Matlab 2016a and it supports only 64bit, and i was building mex for 32bit.

  • Sumit Jagdale

    Hello,
    I have successfully build the project ZED_save_depth.
    I am not able to save neither point cloud nor depth map images.
    Please help…Thanks!

    • Hi,
      It seams that you mixed some code (The cmd line specify Save Depth, whereas the popup tell PCL).
      Be sure to use the appropriate sample, if you can’t fix the error, contact directly the support.

  • Mahé

    Hello, I have these errors message during CMake project creation, anybody knows how to solve it ?
    thx

    CMake Error: The following variables are used in this project, but they are set to NOTFOUND.

    Please set them or make sure they are set and tested correctly in the CMake files:

    CUDA_nppi_LIBRARY (ADVANCED)

    linked by target “mexZED” in directory C:/TRAVAIL/MATLAB/zed-matlab-master/src/mex

    CUDA_npps_LIBRARY (ADVANCED)

    linked by target “mexZED” in directory C:/TRAVAIL/MATLAB/zed-matlab-master/src/mex

    Configuring incomplete, errors occurred!

  • Mahé

    Well, after several problems the files are finally built : yes ! (Windows 7 x64, Matlab R2015, CUDA 7.5)

    I launch the ZED_Camera.m file and get this error message :

    Invalid MEX-file ‘C:TRAVAILMATLABzed-matlab-mastermatlabmexZED.mexw64’: Le module spécifié est introuvable.

    Error in ZED_Camera (line 11)
    result = mexZED(‘init’, 720, ‘quality’)

    Is it a Problem to detect the ZED camera ?

    thank you !

    • Hi,
      please pull the latest version of the code from github and be sure to have the latest SDK (v1.0.0).
      And don’t forget to install the compiled code.

      • Joshua

        Hi,
        I have exactly the same problem as @disqus_NfjlJOvdgF:disqus :

        “Invalid MEX-file ‘D:Documentszed-matlab-master-newmatlabmexZED.mexw64’: The specified module could not be found.

        Error in ZED_Camera (line 11)
        mexZED(‘create’, 720, 60); ”

        I’ve got the latest version of the code and running v1.0.0

        @stereolabs:disqus What do you mean by “don’t forget to install the compiled code”
        Thank you in advance 🙂

        • Jonathan S Matthis

          I am having the exact same problem and error message as Joshua and Mahe. Has anyone figured out a solution?

          I have v 1.0.0. installed and the latest software from Github. Like Joshua, I do not know what you mean by “Don’t forget to install the compile code.”

          Is that what I did when I completed the “Right click on the project named ‘INSTALL’ and select ‘Build’” step in the instructions?

          I’m so close to being able to use this camera as the useful tool it is, please help! Thank you!

          • Mahé

            Hello, finally it works. I use the latest version of the code and run the v1.0.0
            detail :
            Cmake : ok
            VS13 to build all the project : ok (1 project has been skipped, I don’t know why)
            then : copy the “mexZED.mexw64” file located in .zed-matlab-masterbuildmexDebug in the .zed-matlab-mastermatlab directory
            run Matlab
            check the ZED camera has been detected (“webcamlist” should retun ‘ZED’ – and if it’s not the case, then unplug/plug your ZED camera)
            then run the ZED-Camera.m file

            Now, next step for me : I would like to use the Matlab Compiler to export this script to a deployed application, I hope this can work..

  • Jeremy Zhi Huang

    I got the error:

    CMake Error at CMakeLists.txt:36 (find_package):
    By not providing “FindOpenCV.cmake” in CMAKE_MODULE_PATH this project has
    asked CMake to find a package configuration file provided by “OpenCV”, but
    CMake did not find one.

    Could not find a package configuration file provided by “OpenCV” (requested
    version 3.1) with any of the following names:

    OpenCVConfig.cmake
    opencv-config.cmake

    Add the installation prefix of “OpenCV” to CMAKE_PREFIX_PATH or set
    “OpenCV_DIR” to a directory containing one of the above files. If “OpenCV”
    provides a separate development package or SDK, be sure it has been
    installed.

    Do you know how to solve that? I’m using Windows 10 x64bit, and I’ve installed VS2013 and Zed SDK

    • Zhen Xie

      I met the similar problem. you can use the Compiler with x64 just like“Visual Studio 12 2013 win64”

  • Mike

    Hi!
    When I write a simple program such as the one below with an empty main:
    //standard includes
    #include
    #include
    #include

    //opencv includes
    #include
    #include
    #include
    #include

    //ZED Includes
    #include

    using namespace sl::zed;
    using namespace std;

    //Main function

    int main(int argc, char **argv) {

    return 0;
    }

    I get the following error:
    1>mySVOReaderMain.obj : error LNK2019: unresolved external symbol “public: __thiscall sl::zed::Camera::Camera(enum sl::zed::ZEDResolution_mode,float,int)” (??0Camera@zed@sl@@QAE@W4ZEDResolution_mode@12@MH@Z) referenced in function “public: void __thiscall sl::zed::Camera::`default constructor closure'(void)” (??_FCamera@zed@sl@@QAEXXZ)

    Any idea how to fix this?

    • Hi Mike,
      It seems that you didn’t link the library (sl_zed64.lib).

  • HeeYong

    Hi,

    I am using Cmake version(3.6.1) and after configure i got message
    “CMake Warning (dev) in CMakeLists.txt:
    No cmake_minimum_required command is present. A line of code such as

    cmake_minimum_required(VERSION 3.6)

    should be added at the top of the file. The version specified may be lower
    if you wish to support older CMake versions for this project. For more
    information run “cmake –help-policy CMP0000″.
    This warning is for project developers. Use -Wno-dev to suppress it.

    Configuring done

    CMake Error at CMakeLists.txt:14 (add_library):
    Cannot find source file:

    C:/Users/PSD/Desktop/zed-matlab-master/src/mex/Matlabdef.def

    Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
    .hxx .in .txx

    CMake Error: CMake can not determine linker language for target: mexZED”
    What is apropriate version?????

    • Hi HeeYong,
      It seems that you try to compile the CMakeLists.txt from ‘zed-matlabsrcmex’ instead of the one on ‘zed-matlabsrc’,
      Be sure to pull the latest version of the code from https://github.com/stereolabs/zed-matlab and to use the latest version of the ZED SDK.

  • Mahé

    The program is running, I have one question regarding the results obtained.

    I have a differencies on the values (for the same camera position, and focusing on the same pixel) : I have differencies between the value provided by the ‘ZED Depth Viewer.exe’ program and this script.

    for example :
    – the ZED Depth Viewer : gives 1.8 meter for one pixel in the image
    – this matlab scrip gives 1.5 meter

    real value measured with a laser pointer is 1.8 meter.

    So my question is : does this Matlab script use the good calibration file of the camera ? and if it’s not the case : how is it possible to load the good calibration file of the camera with matlab ?

    thanks for your answer.

  • sezaierel@gmail.com

    Hi,

    Can I use any Matlab versiyon or I have to use specific one?

  • Stephen Lewis

    All went well until I opened MEX.sln with Visual Studio (14 2015) and got 1 warning and 1 error:
    Warning: Ignoring unknown option ‘-std=c++0x’
    Error: Cannot open include file: ‘sl/Camera.hpp’: No such file or directory
    I’ve tried adding the directories containing camera.hpp to the additional include directories under C/C++ General in the project properties but this didn’t fix the issue.
    The offending line in mexZED.cpp is:
    // ZED
    #include
    Has anyone encountered this? Thanks.

    • Tarik Loukili

      Hi Stephen, I had the same error.

      so what I did and it worked for me :

      1- I installed the Zed SDK

      2- Load the zed-matlab-master and create a build dir

      3- In CMake GUI I pointed OpenCV_Dir to the opencv that came with the SDK it is located in

      C:Program Files (x86)ZED SDKdependenciesopencv_3.1.0
      4- configure, generate and build solution in visual Studio
      it should work 🙂