1 2 Previous Next 21 Replies Latest reply on Jul 13, 2018 3:02 PM by Intel Corporation

    Mapping Depth to RGB in UWP

    rschu

      We are using the UWP Windows.Media.Capture APIs to access the RealSense D415. We need to map the depth data onto the color for rendering a point cloud.

       

      We are doing a manual registration / mapping based on camera intrinsics and extrinsics. We can get the camera intrinsics from the UWP APIs but the coordinate mapping transform (extrinsics) are not provided.
      This call would normally provide a 4x4 matrix for doing the transformation and it works fine with Kinect v2 but for the RealSense this returns a null value.

      var matDepthToRgb = depthSource.Info.CoordinateSystem.TryGetTransformTo(colorSource.Info.CoordinateSystem)

       

       

      I know there is a point cloud sample in the native RealSense SDK that is using

      environment::get_instance().get_extrinsics_graph()

      But we need to use the UWP Windows.Media.Capture APIs and don't see those extrinsics values exposed. Not via the coordinate system transformation nor another API. 

       

      I have the latest UWP driver and D415 firmware installed as of today.

       

       

      Can you help?

        • 1. Re: Mapping Depth to RGB in UWP
          MartyG

          A couple of months ago, Jesus G from Intel customer support said to another UWP user, "There is no UWP API that is specific to the RealSense cameras.  The purpose of the driver is to allow the cameras to work with any UWP app using the Microsoft APIs."

           

          The full discussion, if you have not seen it already, can be viewed here:

           

          Windows UWP support of D415 and D435

          • 2. Re: Mapping Depth to RGB in UWP
            rosme

            @MartyG, I believe @rschu is running into the same issue that I reported in the forum thread that you linked to.

             

            Here's what I wrote:

            "I also tested the CameraStreamCorrelation example and I ran into an issue where there is no valid CoordinateSystem in the provided colorframe. This will cause the TryCreateCoordinateMapper to fail. That coordinatemapper is needed to map depth pixels from depth space to color space. I suspect that the implementation of this API is still a work in progress."

             

            And here's what Jesus answered:"Thanks for the bug report. I will report it to the team. ..."

             

            So we are eagerly awaiting a fix for this.

            • 3. Re: Mapping Depth to RGB in UWP
              rschu

              Hey Marty, thanks for linking to the other post. I will reply there as well now.

               

              That is exactly the issue with the RealSense UWP driver, it does not provide the proper API mapping. The CoordinateMapper or the CoordinateTransform just return null. They should provide the mapping between the streams like depth to RGB or RGB to depth.

              The data is there in Intel's native SDK with the extrinsics but it's not exposed via the UWP Windows.Media.Capture which is preventing us from having our holographic telepresence solution running with RealSense and therefore we won't buy more RealSense devices for now which could have been a lot for our customers.

              We are hoping Intel will fix that flaw in their UWP driver or provide a workaround in the meantime like a static transformation matrix if the devices are similar manufactured. Right now it's a show stopper.

              • 4. Re: Mapping Depth to RGB in UWP
                MartyG

                I researched your case carefully regarding the possible workaround you mentioned of static transforms.  Apparently, others have been able to publish the static transform of an older R200 RealSense camera model using an instruction in the ROS software called tf_static.  The new SDK 2.0 for the 400 Series cameras is fully compatible with ROS, so I would expect that tf_static ought to be usable on these camera models too.

                 

                The ROS documentation says "it is expected that any transform on this topic can be considered true for all time.  Internally any query for a static transform will return true".

                 

                ROS has a wiki page on the subject.

                 

                http://wiki.ros.org/tf

                • 5. Re: Mapping Depth to RGB in UWP
                  rschu

                  Thanks. I just need the matrix Depth to RGB in plain text so I can hardcode it for now. Do you have it handy?

                   

                  But in fact Intel needs to fix their UWP driver to support proper coordinate mapping otherwise it's not useful at all. I mean Microsoft advertises it as the Kinect replacement so I'd expect Intel and MSFT to work together to provide something equally good.

                  • 6. Re: Mapping Depth to RGB in UWP
                    MartyG

                    There is a script for SDK 2.0 that contains formulas for calculating its distortion and transform, if that is any use to you.

                     

                    librealsense/rsutil.h at 5e73f7bb906a3cbec8ae43e888f182cc56c18692 · IntelRealSense/librealsense · GitHub

                    • 7. Re: Mapping Depth to RGB in UWP
                      rschu

                      Thanks Marty. As you can see in the method there it gets the const struct rs2_extrinsics * extrin passed. See my initial post that this is not exposed in UWP so I can't access it there...

                       

                      Intel, can we get an official comment when UWP driver support will finally provide coordinate mapping?

                      • 8. Re: Mapping Depth to RGB in UWP
                        Intel Corporation
                        This message was posted on behalf of Intel Corporation

                        Hello @rschu and @rosme,

                        Intel and Microsoft are working together on a fix. I will post here when I have an udpate. For now, you can use this workaround in the CameraStreamCorrelation sample, 

                        Please add next lines under FrameRenderer.cpp:126

                           CameraIntrinsics^ cameraIntrinsic =  colorFrame->VideoMediaFrame->CameraIntrinsics;
                            SpatialCoordinateSystem^ spatialCoordinateSystem = colorFrame->CoordinateSystem;

                            if (spatialCoordinateSystem == nullptr)
                            {
                                return;
                            }
                            // Create the coordinate mapper used to map depth pixels from depth space to color space.
                            DepthCorrelatedCoordinateMapper^ coordinateMapper = depthFrame->VideoMediaFrame->DepthMediaFrame->TryCreateCoordinateMapper(cameraIntrinsic, spatialCoordinateSystem);

                        1 of 1 people found this helpful
                        • 9. Re: Mapping Depth to RGB in UWP
                          rschu

                          Thank you Intel! Great to hear.

                           

                          Any ETA when the fix will be in the driver?

                           

                          Also is there a way to get the the 4x4 matrix with a workaround and not use the CoordinateMapper?

                          We are doing a manual registration / mapping based on camera intrinsics and extrinsics. This call would normally provide a 4x4 matrix for doing the transformation:

                          var matDepthToRgb = depthSource.Info.CoordinateSystem.TryGetTransformTo(colorSource.Info.CoordinateSystem)

                          • 10. Re: Mapping Depth to RGB in UWP
                            rschu

                            I tried the suggested workaround but it fails the same way with a null CorrdSys.

                             

                            This hits true and therefore returns:

                                if (spatialCoordinateSystem == nullptr)

                                {

                                    return;

                                }

                             

                            Any ETA when a fix will be delivered? A coarse date would be fine. Like next month or are we talking about H2 of 2018?

                             

                            Thank you again.

                            • 11. Re: Mapping Depth to RGB in UWP
                              Intel Corporation
                              This message was posted on behalf of Intel Corporation

                              Hello rschu,
                               
                              We are working with Microsoft on a fix but we do not have an ETA yet.
                               
                              Best regards,
                               
                              Josh B.
                              Intel Customer Support
                               

                              • 12. Re: Mapping Depth to RGB in UWP
                                rschu

                                Thanks Josh. Appreciate it.

                                 

                                Would be good if you can reply here when it hits. Also, I'm always up to test early bits so if you have a preview version to test the fix, feel free to reach out.

                                • 13. Re: Mapping Depth to RGB in UWP
                                  Intel Corporation
                                  This message was posted on behalf of Intel Corporation

                                  Hello rschu,
                                   
                                  Thank you for your reply.
                                   
                                  We will be letting you know as soon as we have any update.
                                   
                                  Thank you for your patience and understanding.
                                   
                                  Best regards,

                                  Josh B.
                                  Intel Customer Support
                                   

                                  • 14. Re: Mapping Depth to RGB in UWP
                                    rschu

                                    Josh, do you have an update for us? We are desperately waiting for a fix for this.

                                    And again, would be happy to try new beta-state drivers any time.

                                    1 2 Previous Next