Edit: PDF manuals for the R3 SDK can be found in the following folder:

 

C:// Program Files(x86) > Intel > RSSDK > docs > PDF

 

The manuals for particular SDK optional modules seem to be added as each module is installed.

 

Hi everyone,

 

I have been investigating the process for setting up the RealSense camera in the Unity game creation engine with the '2016 R3' SDK.  Up until now, attempts to get it working have failed, because the structure of the R3 SDK is very different to the previous R2 version.  I have made some progress though and will share it in this article, updating it as new discoveries are made.

 

STEP 1

Download the 2016 R3 SDK at this download location and install it:

 

Intel® RealSense™ SDK | Intel® Software

 

1.jpg

 

Download and install the 'Essentials' module first, and then any optional modules from the same page that your project may require (Face Tracking, Hand Trackng, etc).

 

When downloading some of the optional modules such as Face and Hand, you may encounter an error message during installation about not being able to connect to the network to download a particular file.  This is the 'Visual C++ 2015 Redistributable'.  Although this message is alarming, it seems to occur not because the file is missing from the download, but rather because it is already installed on your machine and therefore does not need to be installed again.

 

STEP 2

 

If you have used earlier versions of the RealSense SDK with Unity before then you should be prepared to need to make big changes in your thinking, because the R3 SDK's Unity implementation is completely different, both in folder structure and the names of the DLL library files that make the camera function.

 

Because of this, it is fruitless to try to import RealSense's new Unity implementation into an existing RealSense-equipped project, as you will only get red error messages due to your project being unable to locate the files that it is looking for.  The folder and file changes render the Unity Toolkit files from SDK versions up to 2016 R2 practically useless in their current form.  You will therefore have to start fresh with a brand new Unity project file.

 

STEP 3

 

Open Unity and go to 'File > New Project' to start a new, clean Unity project.

 

2.jpg

 

3.jpg

 

STEP 4

 

The default installation location for the Unity-related files for the RealSense R3 SDK on your computer should be:

 

C: // Program Files (x86) \  Intel \ RSSDK \ framework \ Unity

 

As you can see below, the structure of R3's Unity framework is very different to the framework you may have used in previous SDK versions.

 

4.jpg

 

The files listed in your particular PC's folder will depend on which of the optional modules you downloaded, if any.  There should at least be an 'Intel.Realsense.core' file there, representing the 'Essentials' module, with further files added such as Face and Hand when you install optional modules.

 

STEP 5

 

Double left-click on the 'Intel.Realsense.core' file whilst Unity is open.  This will cause an 'Import Unity Package' window to pop up in Unity.

 

5.jpg

 

Left-click on the 'Import' button at the base of the window to import the files of the 'Essentials' RealSense module into your new Unity project.

 

6.jpg

 

STEP 6

 

Repeat the process for all of the optional module files that you have in your SDK Unity framework folder until they are all imported into the 'RSSDK' folder in your Unity project.

 

7.jpg

 

If you browse through the folders of the optional modules then you can see how much it has changed from the Unity Toolkit of version R2 and earlier.  Instead of only two primary DLL library files to operate the camera, there is now a separate DLL file for each optional module.

 

8.jpg

 

The original familiar library files have also disappeared, replaced in the Core folder by 'Libpxccore_c'.

9.jpg

 

It is clear, then, why attempting to use these files in an existing RealSense-equipped project in Unity that contains the old Unity Toolkit files generates red errors.

 

STEP 7

 

Having completed the setup of our new 2016 R3 SDK project in Unity, you can now run the new project for the first time.  And the result is ... nothing happens to the camera at all, not even a green light.  It is some kind of forward progress from having red errors though!

 

10.jpg

 

Quite simply, because there are no sample scripts provided with the R3 implementation of Unity, the scene does not know what to do with the camera because there are no scripts telling it to activate.

 

Even if a simple cube object is created and a test script placed inside it that contains RealSense camera code, it red-errors because the structure of the RealSense implementation in Unity has changed so much.

 

What will be necessary from this point onwards is to work out how to write scripts that will work with the new R3 structure.

 

It is likely that the new modular structure of the RealSense SDK from '2016 R3' onwards will be the standard set for subsequent RealSense SDK releases in 2017 / 2018, and that 2016 R2 will be obsoleted in the same way that previous SDKs before R2 were.

 

Developers who wish to make use of Unity therefore stand the best chance of future-proofing their applications and making them easier to upgrade by adopting and learning the new modular system now, rather than persisting with the previous Unity implementation that spanned from the launch of RealSense in 2014 to version 2016 R2.

 

Continuing the process of getting started with the 2016 R3 SDK in Unity: although R3 is not supplied with a Unity Toolkit package of tools and sample scripts like in the previous SDKs due to R3's new and very different structure, a sample Unity program is provided - a Unity version of the 'RawStreams' program that regular users of the RealSense SDK will be familiar with.

 

The default installation location for this sample is: C:// Program Files(x86) / Intel / RSSDK / sample / core / RawStreams.Unity

 

1.jpg

This sample is not yet listed in R3's Sample Browser application.  So in order to make it runnable, the folder needs to be copied to a location where data can be saved to it as well as read from it.  The desktop is a suitable location to place it.

 

Right-click on the folder and select the 'Copy' option from the menu.  Then go to the desktop, right-click on it and select the 'Paste' option from the menu to place a copy of the folder there.

 

3.jpg

As usual with RealSense's Unity sample programs, it can be run without setting up a new Unity project by

 

- Starting up Unity

-  Clicking on the 'Open' option on its project selection window

- Browsing to the folder containing the RawStreams.Unity folder, highlighting it and left-clicking the 'Select Folder' option to open the sample in Unity.

 

5.jpg

 

Upon clicking the 'Select Folder' option, you will be notified that the version of Unity that you are opening the sample in is newer than the version that the sample was created in, assuming that your Unity version is newer than 5.2.3.  Left-click the 'Continue' button to proceed with opening the sample.

 

6.jpg

 

Once Unity has updated the sample's files, it will open in its default New Project view.

 

7.jpg

 

We need to load the sample's Scene file into Unity before we can use it.  Left-click on the 'File' menu and select the 'Open Scene' option.  Browse to the RawStreams.unity > Assets > Scenes folder and select the file called 'main'.

 

8.jpg

 

The RawStreams sample project will now load into Unity.

 

9.jpg

 

Left-click on the small triangular 'Play' button at the center-top of the Unity window to run the RawStreams.Unity sample program.

 

Success!  A dual window stream of the RGB and depth cameras is displayed.

 

10.jpg

 

Having successfully run the new RawStreams_Unity sample, we will open its 'RawStreamsController' script in the Unity script editor to learn more about how scripting works in Unity under the 2016 R3 SDK.

 

Left-click on the object in Unity's left-hand Hierarchy panel called 'RawStreamsController' to display its settings in the right-hand Inspector panel, including the 'RawStreamscController' script file that provides the programming for the sample program.

 

1.jpg

 

Left-click on the small gear-wheel icon at the end of the row containing the script's name to open the script's menu, and left-click on the 'Edit Script' menu option to open the script in the Unity script editor.

 

2.jpg

 

using UnityEngine;

using System.Collections;

using Intel.RealSense;

// For each subsequent algorithm module "using Intel.RealSense.AlgorithmModule;"

 

public class RawStreamsController : MonoBehaviour {

 

  [Header("Color Settings")]

  public int colorWidth = 640;

  public int colorHeight = 480;

  public float colorFPS = 30f;

  public Material RGBMaterial;

 

  [Header("Depth Settings")]

  public int depthWidth = 640;

  public int depthHeight = 480;

  public float depthFPS = 30f;

  public Material DepthMaterial;

 

  private SenseManager sm = null;

  private SampleReader sampleReader =  null;

  private NativeTexturePlugin texPlugin = null;

 

  private System.IntPtr colorTex2DPtr = System.IntPtr.Zero;

  private System.IntPtr depthTex2DPtr = System.IntPtr.Zero;

 

  void SampleArrived (object sender, SampleArrivedEventArgs args)

  {

  if(args.sample.Color != null) texPlugin.UpdateTextureNative (args.sample.Color, colorTex2DPtr);

  if(args.sample.Depth != null) texPlugin.UpdateTextureNative (args.sample.Depth, depthTex2DPtr);

  }

 

  // Use this for initialization

  void Start () {

 

  /* Create SenseManager Instance */

  sm = SenseManager.CreateInstance ();

 

  /* Create a SampleReader Instance */

  sampleReader = SampleReader.Activate (sm);

 

  /* Enable Color & Depth Stream */

  sampleReader.EnableStream (StreamType.STREAM_TYPE_COLOR, colorWidth, colorHeight, colorFPS);

  sampleReader.EnableStream (StreamType.STREAM_TYPE_DEPTH, depthWidth, depthHeight, depthFPS);

 

  /* Subscribe to sample arrived event */

  sampleReader.SampleArrived += SampleArrived;

 

  /* Initialize pipeline */

  sm.Init ();

 

  /* Create NativeTexturePlugin to render Texture2D natively */

  texPlugin = NativeTexturePlugin.Activate ();

 

  RGBMaterial.mainTexture = new Texture2D (colorWidth, colorHeight, TextureFormat.BGRA32, false); // Update material's Texture2D with enabled image size.

  RGBMaterial.mainTextureScale = new Vector2 (-1, -1); // Flip the image

  colorTex2DPtr = RGBMaterial.mainTexture.GetNativeTexturePtr ();// Retrieve native Texture2D Pointer

 

  DepthMaterial.mainTexture = new Texture2D (depthWidth, depthHeight, TextureFormat.BGRA32, false); // Update material's Texture2D with enabled image size.

  DepthMaterial.mainTextureScale = new Vector2 (-1, -1); // Flip the image

  depthTex2DPtr = DepthMaterial.mainTexture.GetNativeTexturePtr (); // Retrieve native Texture2D Pointer

 

  /* Start Streaming */

  sm.StreamFrames (false);

 

  }

 

  // Use this for clean up

  void OnDisable () {

 

  /* Clean Up */

  if (sampleReader != null) {

  sampleReader.SampleArrived -= SampleArrived;

  sampleReader.Dispose ();

  }

 

  if (sm != null) sm.Dispose ();

  }

 

}

 

*****************************

 

The header of the script provides our most useful clue about how RealSense camera scripting works in Unity in the R3 SDK.

 

3.jpg

To specify that the script uses the RealSense camera, we must place in the header:

 

using Intel.RealSense;

 

The comment below this line informs us that to access the main and optional feature modules of the R3 SDK, we must use the format:

 

using Intel.RealSense.AlgorithmModule;

 

substituting the word AlgorithmModule for the name of the module.

 

If we revisit the SDK's Unity framework folder then we can find out the module names that the script expects to be provided in the script header if those features are accessed with the script.

 

4.jpg

We can assume that the Essentials (core) module is already referenced in the script as 'using Intel.RealSense', otherwise the script would not be able to function without accessing the Essentials module.  So the algorithms that should be listed are the additional optional ones that are installed in your Unity R3 project.

 

If our script were to use the Face (face) and Hand (hand) algorithms, then our header may look like this:

 

using UnityEngine;

using System.Collections;

using Intel.RealSense;

using Intel.RealSense.Face;

using Intel.RealSense.Hand;

 

However, we provide this information here just for the purposes of learning scripting in Unity with R3, since the Face and Hand optional modules are not used in the RawStreams_Unity sample.

 

Subsequent experimentation revealed that if you do not have the referenced modules installed in your project then the module name will be highlighted in red text to indicate that Unity cannot find the module.

 

Once a module is installed, Unity makes it easy to confirm what the correct algorithm name is by typing the first letter of that algorithm's name after 'using Intel.Realsense.'

 

1.jpg

 

The other interesting detail we can learn from the RawStreamsController script is that the Sense Manager component is still used in Unity in R3, just as it has been with previous versions of the SDK.

 

5.jpg

Finally, if we turn our attention away from the scripting of the sample and look at Unity's 'Assets' panel, a browse through the folder structure of RSSDK and its sub-folders demonstrates the differences between SDK versions R2 and R3 that were shown at the very beginning of this article when setting R3 up in Unity.

 

6.jpg

 

The 'Intel.RealSense' file in the Plugins folder is connected to in the scripting with 'using Intel.Realsense', whist the camera's library file driver in the 'x86_64' folder is called 'libpxccore_c'  (matching the 'core' name of the Essentials module), replacing the familiar pair of library files in the 'Plugins' and 'Plugins_Managed' folders that were used up until the R2 SDK.