Windows UWP support of D415 and D435

[RSmee]

I tested the new D415 and D435 to see if they supported the Windows UWP API and was pleasantly surprised that they do.

However there seem to be a few differences with what I would expect.

1. Each device reports three MediaFrameSourceGroups where I would expect only one per device.

D415:

Intel(R) RealSense(TM) 415 RGB

Intel(R) RealSense(TM) 415 Depth

Intel RS400 Cameras

D435:

Intel(R) RealSense(TM) 430 with RGB Module RGB

Intel(R) RealSense(TM) 430 with RGB Module Depth

Intel RS400 Cameras

The generic RS400 Cameras are the ones that combine the RGB and Depth into the single MediaFrameSourceGroup that I would expect, but I would be better if they carried the specific type in their displayname.

2. Infrared is reported as color

The "Intel RS400 Cameras" reports the support of three MediaFrameSources, two color and one depth.

It turns out that one of the two color MediaFrameSources actually contains the infrared image of the right IR camera (when seen from the front).

I would expect this MediaFrameSource to have the Infrared type.

Is this the best place to report issues like this?

I am eagerly awaiting support for synchronized streams, but I believe we will have to wait for new firmware before this can be tested. <span style="display: inline !important; float: none; background-color: transparent; color: # 3d3d3d; font-family: 'Helvetica Neue',Helvetica,Arial,'Lucida Grande',sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-tr...

[idata]

Hello Rosme,

Thanks for contacting Intel customer support. I am escalating your case to engineering for them to check the information you have provided. Thanks for your patience and understanding. Best Regards, Juan N.

[idata]

Hello Rosme,

I have just received a notification from the engineering team with the following information: "Regarding Issue 1. This is by design. By definition each camera device creates a Sensor Group (SG) on its own. For two cameras (asrc, awgc etc.) on same module (dedicated PID), we add the SG that includes them both. So if we take only depth you'll have a single SG. If depth and color you'll end up with Color SG, Depth SG and Depth+Color SG (that should be used to sync depth and RGB). We are digging into the second issue but it may be due to a Microsoft limitation."Thanks for your patience. Best Regards, Juan N.

[idata]

Hello Rosme,

We received further clarification from the RealSense team on question 2." The image streams from Left and Right Imagers in D400 series cameras (which include D415 and D435) are not just IR and include visible + IR. Y8 is more of grayscale color than IR."Regards,JesusIntel Customer Support

[EThol]

Thank you for the information provided in this thread. I am very interested in this development.

I use the "CameraStreamCorrelation" sample from the Windows-universal-samples (https://github.com/Microsoft/Windows-universal-samples GitHub - Microsoft/Windows-universal-samples: API samples for the Universal Windows Platform.) to test the UWP API with a RealSense D435 camera. But for me the call "MediaFrameSourceGroup::FindAllAsync()" only reports the two items:

Intel(R) RealSense(TM) 430 with RGB Module RGB

Intel(R) RealSense(TM) 430 with RGB Module Depth

but not the "Intel RS400 Cameras" which is said above to contain both RGB and Depth (which is what I need). What do I need to do to get the "Intel RS400 Cameras" group?

Where can I find the latest information and documentation about the UWP API for RealSense D400 cameras?

Thanks,

Erik

Here is my setup:

Windows 10 Pro, Version 1709, OS Build 16299.192

Visual Studio Community 2017, Version 15.5.6

Windows 10 SDK 10.0.16299.0 (as reported by Visual Studio installer)

Intel RealSense 435, Firmware version: 05.08.15.00

[RSmee]

Hi Jesus,

I understand the technical reasoning behind it, but I it makes it harder to select the real color source. Now the RGB and the "visible + IR" camera both report that they are a color source. The Microsoft example CameraFrames (https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/CameraFrames Windows-universal-samples/Samples/CameraFrames at master · Microsoft/Windows-universal-samples · GitHub ) automatically takes the first color source which happens to be the "visible + IR" grayscale color.

kind regards,

Roland

[idata]

Hello Erik,

We just learned that Intel took down the UWP driver from the Windows Update servers on January 28 to make functional changes. This may be while the Windows UWP app isn't working for you. Check your Device Manager if you have entries for the D400 camera under Imaging Devices. If the driver is installed correctly you will see three entries.

Also, there is no UWP API that is specific to the RealSense cameras. The purpose of the driver is to allow the cameras to work with any UWP app using the Microsoft APIs.

Regards,

JesusIntel Customer Support

[RSmee]

Hi Jesus,

I also tested the CameraStreamCorrelation example and I ran into an issue where there is no valid CoordinateSystem in the provided colorframe. This will cause the TryCreateCoordinateMapper to fail. That coordinatemapper is needed to map depth pixels from depth space to color space. I suspect that the implementation of this API is still a work in progress.

kind regards,

Roland

[idata]

Hello Rosme,

Thanks for the bug report. I will report it to the team. Support on the UWP driver is limited now since it has been taken down temporarily.Regards,JesusIntel Customer Support

[RSmee]

" The image streams from Left and Right Imagers in D400 series cameras (which include D415 and D435) are not just IR and include visible + IR. Y8 is more of grayscale color than IR."

Funny that you use this statement to defend that the stream should not be of the infrared type and then in the Unity wrapper simply call it "infrared".

Not really consistent.

https://github.com/IntelRealSense/librealsense/tree/development/wrappers/unity librealsense/wrappers/unity at development · IntelRealSense/librealsense · GitHub

kind regards,

Roland

[RSchu10]

Hey Jesus,

I'm running into the same issues.

Please see my post here for further details:

The RealSense UWP driver does not provide the proper API mapping. The CoordinateMapper or the CoordinateTransform just return null. They should provide the mapping between the streams like depth to RGB or RGB to depth.

The data is there in Intel's native SDK with the extrinsics but it's not exposed via the UWP Windows.Media.Capture which is preventing us from having our holographic telepresence solution running with RealSense and therefore we won't buy more RealSense devices for now which could have been a lot for our customers.

We are hoping Intel will fix that flaw in their UWP driver or provide a workaround in the meantime like a static transformation matrix if the devices are similar manufactured. Right now it's a show stopper.

What is the status on the bug report from 3 months ago?

[idata]

Hello rschu,

To keep issue tracking manageable, let's follow this conversation in your other /message/539681# 539681 thread. As mentioned previously on your thread; "We are working with Microsoft on a fix but we do not have an ETA yet." Best regards, Josh B.Intel Customer Support

[MartyG]

Updated method for RealSense and UWP: