7 Replies Latest reply on Aug 2, 2017 7:06 AM by SilverHair

    Difference between PXCImage view and cv::Mat view for the depth stream

    lucie93

      Hello

       

      I am working with the RealSense R200 camera using Visual Studio & C++.

      I need to use OpenCV to do image processing so I converted the PXCImage * into cv::Mat.

      But, as you can see, the two images are different (like if I am loosing information).

      Does anybody can help me?

      CaptureDepth.PNGCaptureopenCVDepth.PNG

                                                             Render display screenshot                                                                                                                                Imshow display screenshot

       

      Thanks,

      Lucie

        • 1. Re: Difference between PXCImage view and cv::Mat view for the depth stream
          Intel Corporation
          This message was posted on behalf of Intel Corporation

          Hi Lucie,

          Thank you for contacting us.

          We’ve investigated about your case and found two useful links that relate to this case. The first one is a community thread where the user has a similar question and the second one is a guide to use LibRealSense and OpenCV to stream RGB and Depth Data. The latter has a complete example that explains how to use LibRealSense and OpenCV to display both an RGB stream as well as Depth stream into two separate OpenCV created windows.

          https://software.intel.com/en-us/forums/realsense/topic/599415 
          https://software.intel.com/en-us/articles/using-librealsense-and-opencv-to-stream-rgb-and-depth-data 

          Let us know if you find this useful.

          Regards,
          -Sergio A
           

          • 2. Re: Difference between PXCImage view and cv::Mat view for the depth stream
            lucie93

            Hi Sergio A,

             

            Thank you for you answer ,

            Unfortunately, I already looked those two links and I am still blocked. It might be a format problem or I don't know..

             

            This is my code (a mix of what I found on internet), if someone can help me :

            #include "opencv2/core.hpp"

            #include "opencv2/highgui.hpp"

            #include <windows.h>

            #include "pxcsensemanager.h"

            #include "pxcmetadata.h"

            #include "util_cmdline.h"

            #include "util_render.h"

            #include <conio.h>

            #include <iostream>

            using namespace std;

            using namespace cv;

             

            void RecordORPlayback(pxcCHAR *file, bool record) {

              // Create a SenseManager instance

              PXCSenseManager *sm = PXCSenseManager::CreateInstance();

              PXCSession *session;

              // Set file recording or playback

              sm->QueryCaptureManager()->SetFileName(file, record);

              // Select the color stream

              sm->EnableStream(PXCCapture::STREAM_TYPE_DEPTH, 320, 240, 30);

              // Initialize and Record 300 frames

              sm->Init();

              for (int i = 0; i<100; i++) {

              // This function blocks until a color sample is ready

              if (sm->AcquireFrame(true)<PXC_STATUS_NO_ERROR) break;

              // Retrieve the sample

              PXCCapture::Sample *sample = sm->QuerySample();

              // Go fetching the next sample

              sm->ReleaseFrame();

              }

              // close down

              sm->Release();

            }

             

            cv::Mat PXCImage2CVMat(PXCImage *pxcImage, PXCImage::PixelFormat format)

            {

              PXCImage::ImageData data;

              pxcImage->AcquireAccess(PXCImage::ACCESS_READ, format, &data);

              int width = pxcImage->QueryInfo().width;

              int height = pxcImage->QueryInfo().height;

              if (!format)

              format = pxcImage->QueryInfo().format;

              int type;

              if (format == PXCImage::PIXEL_FORMAT_Y8)

              type = CV_8UC1;

              else if (format == PXCImage::PIXEL_FORMAT_RGB24)

              type = CV_8UC3;

              else if (format == PXCImage::PIXEL_FORMAT_DEPTH_F32)

              type = CV_32F;

              cv::Mat ocvImage = cv::Mat(cv::Size(width, height), type, data.planes[0], data.pitches[0]);

              pxcImage->ReleaseAccess(&data);

              return ocvImage;

            }

             

             

            int main(int argc, WCHAR* argv[]) {

              //Define some parameters for the camera

              cv::Size frameSize = cv::Size(320, 240);

              float frameRate = 30;

              cv::Mat frameColor = cv::Mat::zeros(frameSize, CV_8UC3);

              cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_32F);

             

              /* Creates an instance of the PXCSenseManager */

              PXCSenseManager *pp = PXCSenseManager::CreateInstance();

              if (!pp) {

              wprintf_s(L"Unable to create the SenseManager\n");

              return 3;

              }

             

              /* Sets file recording or playback */

              PXCCaptureManager *cm = pp->QueryCaptureManager();

              pxcCHAR *filename = (L"DEPTHRenderTEST");

              RecordORPlayback(filename, true);

             

              // Create stream renders

              UtilRender renderc(L"Color"), renderd(L"Depth"), renderi(L"IR"), renderr(L"Right"), renderl(L"Left");

              pxcStatus sts;

             

              do {

              /* Apply command line arguments */

              pxcBool revert = false;

              PXCVideoModule::DataDesc desc = {};

              desc.deviceInfo.streams = PXCCapture::STREAM_TYPE_COLOR | PXCCapture::STREAM_TYPE_DEPTH;

              revert = true;

              pp->EnableStreams(&desc);

             

              /* Initializes the pipeline */

              sts = pp->Init();

              if (sts<PXC_STATUS_NO_ERROR) {

              if (revert) {

              /* Enable a single stream */

              pp->Close();

              pp->EnableStream(PXCCapture::STREAM_TYPE_DEPTH);

              sts = pp->Init();

              if (sts<PXC_STATUS_NO_ERROR) {

              pp->Close();

              pp->EnableStream(PXCCapture::STREAM_TYPE_COLOR);

              sts = pp->Init();

              }

              }

              if (sts<PXC_STATUS_NO_ERROR) {

              wprintf_s(L"Failed to locate any video stream(s)\n");

              pp->Release();

              return sts;

              }

              }

             

             

              /* Reset all properties */

              PXCCapture::Device *device = pp->QueryCaptureManager()->QueryDevice();

              device->ResetProperties(PXCCapture::STREAM_TYPE_ANY);

             

              for (;;) {

              // for (int i=0;i<51;i++){

              /* Waits until new frame is available and locks it for application processing */

              sts = pp->AcquireFrame(true);

              if (sts<PXC_STATUS_NO_ERROR) {

              if (sts == PXC_STATUS_STREAM_CONFIG_CHANGED) {

              wprintf_s(L"Stream configuration was changed, re-initilizing\n");

              pp->Close();

              }

              break;

              }

             

              const PXCCapture::Sample *sample = pp->QuerySample();

              if (sample) {

              if (sample->depth && !renderd.RenderFrame(sample->depth)) break;

             

              if (sample->depth) {

              frameDepth = PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH_F32);

              /* string result_depthCV = "DepthOpenCV.png";

              imwrite(result_depthCV, frameDepth);*/

              namedWindow("Depth", CV_WINDOW_NORMAL);

              imshow("Depth", frameDepth);

              waitKey(1);

              }

             

              if (sample->color) {

              frameColor = PXCImage2CVMat(sample->color, PXCImage::PIXEL_FORMAT_RGB24);

              namedWindow("Color", CV_WINDOW_NORMAL); //works without this line too

              imshow("Color", frameColor);

              waitKey(1);

              }

             

              /* Releases lock so pipeline can process next frame */

              pp->ReleaseFrame();

              }

              if (_kbhit()) { // Break loop

              int c = _getch() & 255;

              if (c == 27 || c == 'q' || c == 'Q') break; // ESC|q|Q for Exit

              }

              }

              } while (sts == PXC_STATUS_STREAM_CONFIG_CHANGED);

             

              wprintf_s(L"Exiting\n");

             

              // Clean Up

              pp->Release();

              system("Pause");

              return 0;

            }

             

             

            Thanks,

            Lucie

            • 3. Re: Difference between PXCImage view and cv::Mat view for the depth stream
              Intel Corporation
              This message was posted on behalf of Intel Corporation

              Hi Lucie,

              We looked online and even though we didn’t find any official guides that might help you with the code, we did found a couple of links that may be helpful
              https://software.intel.com/en-us/forums/realsense/topic/624897 
              http://stackoverflow.com/questions/37266390/how-to-convert-realsense-rgb-frame-to-cvmat-in-ubuntu-or-other-linux-env 

              From the looks of both pictures, they’re obviously different, but, the depth patterns don’t seem to be so different from each other.
              It looks as if one of the images needs further calibration in order to look the same as the first image.

              I looked at some examples and found that frameColor and frameDepth used this way solved the issue for some customers: 
              cv::Mat frameColor = cv::Mat::zeros(resolutionColor.height, resolutionColor.width, CV_8UC3);
              cv::Mat frameDepth = cv::Mat::zeros(resolutionDepth.height, resolutionDepth.width, CV_32FC1);

              Keep us updated.

              -Sergio A
               

              • 4. Re: Difference between PXCImage view and cv::Mat view for the depth stream
                Intel Corporation
                This message was posted on behalf of Intel Corporation

                Hi Lucie,

                Do you still need assistance with this case? We’ll be waiting for your response.

                -Sergio A
                 

                • 5. Re: Difference between PXCImage view and cv::Mat view for the depth stream
                  lucie93

                  Hi Sergio,

                   

                  Thanks for your interest in my issue.

                  I just solved it

                   

                  Finally, I used :

                  cv::Mat frameDepth = cv::Mat::zeros(frameSize, CV_16U);

                  cv::Mat frameDepth8b = cv::Mat::zeros(frameSize, CV_8UC1);

                   

                  and then,

                   

                  if (sample->depth) {

                    frameDepth = PXCImage2CVMat(sample->depth, PXCImage::PIXEL_FORMAT_DEPTH);

                    frameDepth.convertTo(frameDepth8b, CV_8UC1, 255.0 / 4096.0);

                   

                  // imwrite

                    string result_depthCV = "DepthOpenCV.png";

                    imwrite(result_depthCV, frameDepth8b, { CV_IMWRITE_PNG_STRATEGY_DEFAULT });

                   

                  // imshow

                    namedWindow("Depth", CV_WINDOW_AUTOSIZE);

                    imshow("Depth", frameDepth8b);

                    waitKey(1);

                  }

                   

                  I don't really understand with I have to convert frameDepth in CV_8UC1 because normally imwrite can save an 16-bit image in PNG format. But it now at least, it works

                   

                  Thanks again ,

                  Lucie

                  • 6. Re: Difference between PXCImage view and cv::Mat view for the depth stream
                    Intel Corporation
                    This message was posted on behalf of Intel Corporation

                    Hi Lucie,

                    Thank you for confirming you’ve fixed the issue and sharing your results with the community. Don’t hesitate to come back if help is needed, we’d be happy to assist you.

                    Regards,
                    -Sergio A
                     

                    • 7. Re: Difference between PXCImage view and cv::Mat view for the depth stream
                      SilverHair

                      Hi Lucie

                      You said you have solved this problem, but when I used your method, it didn't work. As you can see:

                      It is still different.

                      I hope I will receive your reply.