6 Replies Latest reply on Oct 15, 2018 10:27 PM by akclucc

    How to iterate over pointcloud for color data

    akclucc

      I am attempting to access the color values of pixels from the RGB sensor with the goal of storing a few frames as "snapshots" in a bitmap format. Because there is no native .getColor() method available in the SDK like there is for depth, my reading has lead me to create a PCL pointcloud from aligned frames from which to extract RGB values point by point.

       

      I do not understand how to iterate over the pointcloud so that a bitmap in (x,y) could be "colored in" with matching values from cloud->points[i] . How should I approach this next step?

       

      Also, if there is an easier way to access pixel (x,y) from the RGB sensor, please let me know!

        • 1. Re: How to iterate over pointcloud for color data
          akclucc

          It is probably a good idea to show exactly how I am creating the pointcloud:

           

          pcl::PointCloud<pcl::PointXYZRGB> *points_to_pcl(const rs2::points& points, const rs2::video_frame& color){

           

          pcl::PointCloud<pcl::PointXYZRGB> *cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

           

          auto sp = points.get_profile().as<rs2::video_stream_profile>();

          cloud->width = sp.width();

          cloud->height = sp.height();

          cloud->is_dense = false;

          cloud->points.resize(points.size());

           

          auto tex_coords = points.get_texture_coordinates();

          auto vertices = points.get_vertices();

           

          for (int i = 0; i < points.size(); ++i)

          {

          cloud->points[i].x = vertices[i].x;

          cloud->points[i].y = vertices[i].y;

          cloud->points[i].z = vertices[i].z;

           

          std::tuple<uint8_t, uint8_t, uint8_t> current_color;

          current_color = get_texcolor(color, tex_coords[i]);

           

          cloud->points[i].r = std::get<0>(current_color);

          cloud->points[i].g = std::get<1>(current_color);

          cloud->points[i].b = std::get<2>(current_color);

          }

           

          return cloud;

          }

           

          This is adapted from Yonatan-Sade's solution...

          https://github.com/IntelRealSense/librealsense/issues/1601#issuecomment-387512731

          • 2. Re: How to iterate over pointcloud for color data
            akclucc

            So I realized in looking at that code more closely that I could skip the PCL entirely, so I am now running this:

             

            //for reference, myFrame is an EasyBMP bitmap

             

            void rsFrameManager::makeBitmap(const rs2::points& points, const rs2::video_frame& color){

             

            auto sp = points.get_profile().as<rs2::video_stream_profile>();

            auto tex_coords = points.get_texture_coordinates();

            auto vertices = points.get_vertices();

             

            for (int i = 0; i < points.size(); ++i)

            {

            std::tuple<uint8_t, uint8_t, uint8_t> current_color;

            current_color = get_texcolor(color, tex_coords[i]);

             

            int w = (tex_coords[i].u)*width;

            int h = (tex_coords[i].v)*height;

             

            myFrame(w, h)->Red = std::get<0>(current_color);

            myFrame(w, h)->Green = std::get<1>(current_color);

            myFrame(w, h)->Blue = std::get<2>(current_color);

            myFrame(w, h)->Alpha = 0;

            }

            }

             

            Unfortunately, when I print the resulting bitmap, it is just a white frame, so this still does not work, but I suspect this is a step towards my solution.

            • 4. Re: How to iterate over pointcloud for color data
              akclucc

              Thank you for the reply!

               

              I was familiar with the first link you mentioned but not the second. I think I understand the solution used in the second link but am confused as to what color_intrinsic , used to calculate colorLocation, is and how I should calculate it on my end. How does color_intrinsic work?

               

              EDIT:

               

              I have defined color_intrinsic as

               

              rs2_intrinsics color_intrinsic = colorFrame.get_profile().as<rs2::video_stream_profile>().get_intrinsics();

               

              which I believe is proper. Now I am just trying to figure out how to map the verticies to pixels in a BMP and set the colors accordingly. Suggestions?

              • 5. Re: How to iterate over pointcloud for color data
                MartyG

                If you are mapping 3D XYZ vertices to 2D pixels, I believe that the appropriate instruction is rs2_project_point_to_pixel.  There is a detailed discussion about it in the link below.

                 

                How to project points to pixel? Librealsense SDK 2.0

                • 6. Re: How to iterate over pointcloud for color data
                  akclucc

                  Interesting. I was in a time crunch so I used writing to a .png and then converting to .bmp as a bandaid solution to get me up and running for my deadline, but obviously I want to replace that mechanism for performance reasons soon. I will take a look at rs2_project_point_to_pixel this weekend with hopes that it can resolve my current issue on the pointcloud side, where the resulting bitmap only consists of white pixels. I will update once I've gotten to play with this instruction.