Skip navigation

Intel® NUC

5 posts

TechBubble IoT JumpWay Docs

Introduction

 

Here you will find a sample application for TASS PVL, a Computer Vision security system using Intel® Computer Vision SDK and an Intel® Edison connected to the Internet of Things via TechBubble Technologies IoT JumpWay.

 

Once you understand how it works you are free to modify the app accordingly.

 

This project uses two applications:

 

  1. A Windows Computer Vision application.
  2. A Node JS application on an Intel® Edison that receives commands to activate LEDs and a buzzer when known or unknown faces are detected

 

Software requirements

 

  1. TechBubble IoT JumpWay Node JS MQTT Client Library

  2. TechBubble IoT JumpWay WebSocket MQTT Client

  3. Intel® Computer Vision SDK for Windows 10

  4. Microsoft Vcpkg, Paho, Json

  5. Node JS

  6. Visual Studio 2017

 

Hardware requirements

 

  1. Windows PC with 6th Generation Intel® Core™ Processors with Intel® Iris® Pro Graphics and HD Graphics, In our example we are using an Intel® NUC NUC7i7BNH with Intel® OPtane Memory.

  2. 1 x Intel® Edison

  3. 1x Grove starter kit plus - Intel IoT Edition for Intel® Edison

  4. 1 x Blue LED (Grove)

  5. 1 x Red LED (Grove)

  6. 1 x Buzzer (Grove)

  7. 1 x Webcam

 

Before You Begin

 

There are a few tutorials that you should follow before beginning, especially if it is the first time you have used the TechBubble IoT JumpWay Developer Program. If you do not already have one, you will require a TechBubble IoT JumpWay Developer Program developer account, and some basics to be set up before you can start creating your IoT devices. Visit the following IoT JumpWay Developer Program Docs (5-10 minute read/setup) and check out the guides that take you through registration and setting up your Location Space, Zones, Devices and Applications (About 5 minutes read).

 

Preparing Your Windows Device

 

  • Install Intel® Computer Vision SDK

  • Install Microsoft Vcpkg, Paho, Json

  • Install Visual Studio 2017

  • Install Paho MQTT

    C:\src\vcpkg> vcpkg install paho-mqtt:x64-windows

    Once installed, edit the MQTTAsync.h and MQTTClient.h files in C:\src\vcpkg\installed\x64-windows\include.

    Change:

    #if defined(WIN32) || defined(WIN64)#define DLLImport __declspec(dllimport)     #define DLLExport __declspec(dllexport) #else     #define DLLImport extern     #define DLLEx port  __attribute__ ((visibility ("default"))) #endif
  • To:

    #if defined(_WIN32) || defined(_WIN64)     #define DLLImport __declspec(dllimport)     #define DLLExport __declspec(dllexport) #else     #define DLLImport extern     #define DLLExport  __attribute__ ((visibility ("default"))) #endif
  • Install Nlohmann Json

    C:\src\vcpkg> vcpkg install nlohmann-json:x64-windows
  • Plug In Your Webcam

    Plug in your webcam and make sure that you have all of the relevant drivers installed on your machine to recognize the device.

 

Cloning The Repo

 

You will need to clone this repository to a location on your Intel® Edison. Navigate to the directory you would like to download it to and issue the following command, or use the Windows GitHub GUI.

C:\YourChosenLocation> git clone https://github.com/TechBubbleTechnologies/IoT-JumpWay-Intel-Examples.git

IoT JumpWay Device Connection Credentials & Settings

 

  • Follow the TechBubble Technologies IoT JumpWay Developer Program (BETA) Location Application Doc to set up your IoT JumpWay Location Application.

  • Setup an IoT JumpWay Location Device for TASS PVL, ensuring you set up you camera node,as you will need the ID of the camera for the project to work. Once your create your device, make sure you note the MQTT username and password, the device ID and device name exactly, you will also need the zone and location ID. You will need to edit your device and add the rules that will allow it to communicate autonomously with the Intel Edison, but for now, these are the only steps that need doing at this point.

 

Follow the TechBubble Technologies IoT JumpWay Developer Program (BETA) Location Device Doc to set up your devices.

 

IoT JumpWay Device Creation Docs

 

  • Locate and update the following code in TASS-PVL-Windows-Console.cpp, and replace with your device settings.
          int IntelliLanLocation = 0;          int IntelliLanZone = 0;          int IntelliLanDevice = 0;          int IntelliLanSensor = 0;          std::string IntelliLanDeviceN = "YourIoTJumpWayDeviceNameHere";          std::string IntelliLanDeviceU = "YourIoTJumpWayDeviceUsernameHere";         std::string IntelliLanDeviceP = "YourIoTJumpWayDevicePasswordHere";
  • You may also need to edit this value, this allows the application to connect to your webcam, generally the value is either 1 or 0, in my case it is 1 which is the default setting in the provided application. If you receive an error on startup that the application cannot connect to your camera, you will need to modify this setting and ensure that you have installed all the required drivers for your camera.
   int camera = 1;

Additional Include Directories & Library Directories

 

If you installed the Intel® Computer Vision SDK to any directory other than the default (C:\Intel\CV SDK) you will need to update the Additional Include Directories & Library Directories settings to reflect your installed location.

 

Additional Include Directories

Additional Library Directories

Setting Up Your Intel® Edison

 

IoT JumpWay Intel® Edison Dev Kit IoT Alarm

 

The next step is to set up your Intel® Edison so that TASS PVL can communicate with it via the IoT JumpWay. For this, we already created a tutorial, the IoT JumpWay Intel® Edison Dev Kit IoT Alarm, that will guide you through this process. The only difference is that you do not need to set up the Python commands application, as in this project, TASS PVL will replace the Python commands application, to save time, please only follow the steps for the Intel® Edison device Node JS application.

 

You will find the tutorial on the following link:

 

IoT JumpWay Intel® Edison Dev Kit IoT Alarm

 

Once you have completed that tutorial and have your device setup, return he to complete the final integration steps.

 

Setting Up Your Rules

 

You are now ready to take the final steps, at this point you should have everything set up and your Intel® Edison Dev Kit IoT Alarm should be running and connected to the IoT JumpWay waiting for instructions.

 

Next we are going to set up the rules that allow TASS PVL to control your Intel® Edison Dev Kit IoT Alarm autonomously. Go back to the TAS PVL device page and make sure you are on the edit page. Scroll down to below where you added the camera node and you will see you are able to add rules.

 

IoT JumpWay Intel® Edison Dev Kit IoT Alarm

 

The rules that we want to add are as follows:

 

  1. When a known person is identified, turn on the blue LED.

  2. When an unknown person is identified, turn on the red LED.

  3. When an unknown person is identified, turn on the buzzer.

 

The events are going be triggered by warning messages sent from TASS PVL, so in the On Event Of drop down, select WARNING. Then you need to select the camera node you added to the TASS PVL device, as this is the sensor that the warning will come from. Next choose RECOGNISED in the With Warning Of, which will mean that the rule will be triggered when the IoT JumpWay receives a warning message that a known person has been identified, then select the Send Device Command for the Take The Following Action section, choose the Intel® Edison as the device, the blue LED as the sensor, toggle as the action and on as the command. This will then tell the Edison to turn on the blue light in the event of a known person being detected.

 

You should repeat these steps for the red LED and buzzer for the event of NOT RECOGNISED to handle events where an intruder, or unknown person is identified.

 

Ready To Go!

 

And that is it, if you have followed the tutorials correctly, you are now ready to fire up your new security system. Run the Windows console application to begin, and you will see the console window open up and a live stream of your camera, complete with bounding box and emotion status if happy. To train a known user, you simply need them to stand in front of the camera and click R to register their face, and S to save.

 

Viewing Your Data

 

When the program detects a known user or intruder, it will send sensor and warning data for the device it was captured from to the TechBubble IoT JumpWay. You will be able to access the data in the TechBubble IoT JumpWay Developers Area. Once you have logged into the Developers Area, visit the TechBubble IoT JumpWay Location Devices Page, find your device and then visit the Warnings & Sensor Data pages to view the data sent from the application.

 

IoT JumpWay Sensor Data

IoT JumpWay Warning Data

IoT JumpWay Intel® Computer Vision SDK Bugs/Issues

 

Please feel free to create issues for bugs and general issues you come across whilst using the IoT JumpWay Intel® Examples. You may also use the issues area to ask for general help whilst using the IoT JumpWay Intel® Examples in your IoT projects.

 

IoT JumpWay Intel® Computer Vision SDK Contributors

 

Adam Milton-Barker,  Intel Software Innovator

oT JumpWay Intel® NUC DE3815TYKE OpenFace Computer Vision Example

Introduction

Want to turn your Intel® NUC DE3815TYKE, or any Linux box running Ubuntu into an Artificially Intelligent, IoT Connected CCTV Hub? This tutorial is for you!.

What Will We Build?

The tutorial will use TechBubble Technologies IoT JumpWay Python MQTT Library for communication, an Intel® NUC DE3815TYKE or any Linux Desktop running Ubuntu, 1 or more IP Cameras, an optional Realsense camera, and our own deep learning neural network based on the popular OpenFace facial recognition toolkit.

Python Versions

  • 2.7

Software requirements

  1. TechBubble IoT JumpWay Python MQTT Client
  2. OpenFace
  3. Pyrealsense

Hardware requirements

  1. Intel® NUC DE3815TYKE (Should work with other versions of Intel® NUC, or any Linux box with Ubuntu).
  2. 1 or more IP cameras. (Tested with 1)
  3. 1 x Realsense camera (OPTIONAL) (Tested with F200 & R200)

Before You Begin

There are a few tutorials that you should follow before beginning, especially if it is the first time you have used the TechBubble IoT JumpWay Developer Program. If you do not already have one, you will require a TechBubble IoT JumpWay Developer Program developer account, and some basics to be set up before you can start creating your IoT devices. Visit the following IoT JumpWay Developer Program Docs (5-10 minute read/setup) and check out the guides that take you through registration and setting up your Location Space, Zones, Devices and Applications (About 5 minutes read).

 

Preparing Your Intel® NUC DE3815TYKE

If you do not already have Ubuntu Server 16.04 LTS installed on your Intel® NUC DE3815TYKE, follow the Installing Ubuntu Server 16.04 LTS on Intel® NUC DE3815TYKE guide.

 

Cloning The Repo

You will need to clone this repository to a location on your Intel® NUC DE3815TYKE. Navigate to the directory you would like to download it to and issue the following commands.$ git clone https://github.com/TechBubbleTechnologies/IoT-JumpWay-Intel-Examples.git

 

Install Requirements

$ cd IoT-JumpWay-Intel-Examples/Intel-Nuc/DE3815TYKE/Computer-Vision/Python/OpenFace 
$ sudo apt install python-pip
$ pip install --upgrade pip
$ sudo pip install -r requirements.txt

Install Remaining Required Software

  1. Follow the Installing OpenFace on Intel® NUC DE3815TYKE guide.
  2. Follow the Installing Librealsense & Pyrealsense on Intel® NUC DE3815TYKE guide. (OPTIONAL BUT REQUIRED IF USING REALSENSE CAMERA)

IoT JumpWay Device / Application Connection Credentials & Settings

IoT JumpWay Device Creation Docs

  • Retrieve your connection credentials and update the config.json file with your new connection credentials and camera IDs, add a new entry to CameraList for each IP cam, and add your Realsense camera ID.

 

"CameraList": [
  {
  "camID": Your Camera ID from the IoT JumpWay here,
  "camSensorID": Your Camera Sensor ID from the IoT JumpWay here,
  "camZone": Your Camera Zone ID from the IoT JumpWay here,
  "camURL": "URL to your IP camera here"
  }
  ],
  "RealsenseCam": {
  "localCamID": 0,
  "camID": Your Realsense Camera ID from the IoT JumpWay here,
  "camSensorID": Your Realsense Camera Sensor ID from the IoT JumpWay here,
  "camZone": Your Realsense Camera Zone ID from the IoT JumpWay here
  }

     "IoTJumpWayMQTTSettings": {
        "host": "https://iot.techbubbletechnologies.com",
        "port": "8883",
        "SystemLocation": Your Location ID from the IoT JumpWay here,
        "SystemApplicationID": Your Application ID from the IoT JumpWay here,
        "SystemApplicationName": "Your Application Name Here",
        "applicationUsername": "Your Application MQTT Username Here",
        "applicationPassword": "Your Application MQTT Password Here"
    }

 

Preparing Training Data For Your Neural Network

To help solve the open set recognition issue (See KNOWN ISSUES below), we have provided 500 images from the Labeled Faces in the Wild Dataset, this has been tested in small environments, but may not be an accurate solution in larger environments where there is more chance of someone resembling one of the unknown photos. All that remains for you to do is to collect your training data and add it to the training folder ready for training.

Create 1 or more folders in the training/1 directory, these folders will represent classes, and there should be 1 folder / class per person, name the folder using something that will allow you identify who the photos are of, the name of the folder / class will be used by the program to let you know who it has detected. You can use names, user IDs or anything you like for the folder / class names, but bear in mind privacy. We have successfully tested with 10 training images per class, but your application may need more or less than this.

Training Your Neural Network

Now you have added your training data, you should train your neural network, navigate to the root of the project and execute the following command:

$ python TassTrain.py 

Testing Your Neural Network

Before we go any further, we can now test your trained neural network. Add 1 or more images of each person / class you added to the training data into the testing directory, you should try with images that you did not use in the training data. Navigate to the root of the project and execute the following command:

$ python TassTest.py 

If the images you add to the testing directory are not classified correctly after running TassTest.py, you should re train with more examples.

Using 1 Or More IP Cameras

Ensure you have set up your IP camera devices and added the settings correctly to your config.json as mentioned above, then navigate to the root of the project and execute the following command:

$ python TassCore.py 

The program will initiate and your live streams will be processed. First the program will detect if there is a face present in the frame, and then will send the frame through the neural network to detect if it is an intruder or a known person. For trouble shooting, see KNOWN ISSUES and TROUBLE SHOOTING below.

Using A Realsense Camera

Ensure you have set up your Realsense camera device and adding the settings correctly to your config.json as mentioned above, then navigate to the root of the project and execute the following command:

$ python TassRealsense.py 

The program will initiate and your live stream will be processed. First the program will detect if there is a face present in the frame, and then will send the frame through the neural network to detect if it is an intruder or a known person. For trouble shooting, see KNOWN ISSUES and TROUBLE SHOOTING below.

Seeing What Your Neural Network Sees

In the event that a face is detected, the frame will be saved in the frames folder, bounding boxes will be drawn around all faces that are detected and the frames will be placed into either detected or notdetected directories grouped by date, in addition, frames that have faces detected in them will be stored in the identified / notidentified directories of the respective date and time directories, these frames will not have bounding boxes, so you can use them to retrain your model. You can access these images by connecting to your NUC with SFTP and downloading them to your computer.

Viewing Your Data

When the program detects a known user or intruder, it will send sensor and warning data for the device it was captured from to the TechBubble IoT JumpWay. You will be able to access the data in the TechBubble IoT JumpWay Developers Area. Once you have logged into the Developers Area, visit the TechBubble IoT JumpWay Location Devices Page, find your device and then visit the Warnings & Sensor Data pages to view the data sent from the application.

IoT JumpWay Sensor Data

IoT JumpWay Warning Data

KNOWN ISSUES

  1. The Open Set Recognition Issue: The Open Set Recognition Issue is where a neural network will identify someone that it has not been trained on, as someone that it has. In this version of TASS we have seemed to have solved this issue with the use of an unknown class consisting of 500 images of random people from the LFW dataset. In larger environments, this may not solve this issue, but in small environments such as a home or office it should.
  2. Lighting: Lighting is unfortunately quite a large problem that we have not been able to solve as of yet. We find we have best results when there is bright light in front of the face.

TROUBLE SHOOTING

  1. If the OpenCV or Dlib do not detect known faces, it is likely that you have a lighting issue, play with the lighting and test in different scenarios.
  2. If you see Unable To Classify Frame, it means the Open CV or Dlib were able to detect a face, but the classifier was not able to classify the person. There could be two reasons for this, the need for more training, or lighting. You will find the non classified images in the frames directory, there will be a directory for each hour of the day in this directory, look for the notidentified directory and download the images, add them to your training data for the relevant person / class, and retrain the model.
  3. If you see the classifier was unable to classify a known person, the reasons for this could be the same as in point 2 above, repreat the steps provided in that point and see if it helps.

ACKNOWLEDGEMENTS

Parts of this project use code from the following projects:

We would like to acknowledge and thank the creators of these projects.

 

LICENSE

Copyright 2016, Adam Milton-Barker, TechBubble Technologies, All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

View this project on Intel® DevMesh

IoT JumpWay Intel® NUC DE3815TYKE Examples Bugs/Issues

Please feel free to create issues for bugs and general issues you come across whilst using the Intel® NUC DE3815TYKE Docs. You may also use the issues area to ask for general help whilst using the IoT JumpWay Intel® Examples in your IoT projects.

IoT JumpWay Intel® NUC DE3815TYKE Examples Contributors

Adam Milton-Barker,  Intel Software Innovator

 

 

TechBubble IoT JumpWay Docs

Introduction

The following information will help you install Librealsense & Pyrealsense on the Intel® Nuc DE3815TYKE.

Librealsense

$ sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade  
$ sudo apt-get install libusb-1.0-0-dev pkg-config 
$ sudo apt-get install libglfw3-dev 
$ git clone https://github.com/IntelRealSense/librealsense.git 
$ cd librealsense 
$ mkdir build && cd build 
$ cmake ../  $ make && sudo make install

Pyrealsense

$ sudo pip install pycparser  
$ sudo pip install cython 
$ sudo sudo pip install pyrealsense

IoT JumpWay Intel® NUC DE3815TYKE Bugs/Issues

Please feel free to create issues for bugs and general issues you come across whilst using the IoT JumpWay Intel® Examples. You may also use the issues area to ask for general help whilst using the IoT JumpWay Intel® Examples in your IoT projects.

IoT JumpWay Intel® NUC DE3815TYKE Examples Document Contributors

Adam Milton-Barker,  Intel® Software Innovator

TechBubble IoT JumpWay Docs

Introduction

The following information will help you install OpenFace on the Intel® Nuc DE3815TYKE.

OpenFace Prerequisites

$ sudo apt install git  
$ git clone https://github.com/cmusatyalab/openface.git  --recursive

OpenCV 2.4.11

$ sudo apt-get install build-essential  
$ sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev 
$ sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev 
$ wget https://github.com/Itseez/opencv/archive/2.4.11.zip 
$ sudo apt install unzip 
$ unzip 2.4.11.zip 
$ cd ~/opencv-2.4.11 && mkdir release && cd release 
$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. 
$ make  $ sudo make install

Install DLib

$ sudo apt-get install libopenblas-dev liblapack-dev  
$ sudo apt-get install libboost-all-dev 
$ mkdir -p ~/src && cd ~/src 
$ wget https://github.com/davisking/dlib/releases/download/v18.16/dlib-18.16.tar.bz2 
$ tar xf dlib-18.16.tar.bz2 
$ cd dlib-18.16/python_examples 
$ mkdir build && cd build 
$ cmake ../../tools/python 
$ cmake --build . --config Release 
$ sudo cp dlib.so /usr/local/lib/python2.7/dist-packages

Install Torch

$ git clone https://github.com/torch/distro.git ~/torch --recursive  
$ cd ~/torch; bash install-deps; 
$ ./install.sh 
$ source ~/.bashrc 
$ for NAME in dpnn nn optim optnet csvigo cutorch cunn fblualib torchx tds; do luarocks install $NAME; done

OpenFace Installation

$ cd ~/openface  
$ sudo python2 setup.py install

IoT JumpWay Intel® NUC DE3815TYKE Bugs/Issues

Please feel free to create issues for bugs and general issues you come across whilst using the IoT JumpWay Intel® Examples. You may also use the issues area to ask for general help whilst using the IoT JumpWay Intel® Examples in your IoT projects.

IoT JumpWay Intel® NUC DE3815TYKE Examples Document Contributors

Adam Milton-Barker,  Intel® Software Innovator

TechBubble IoT JumpWay Docs

Introduction

The following information will help you install Ubuntu Server 16.04 LTS on the Intel® Nuc DE3815TYKE.

Create Bootable USB On Windows

  1. Download the Ubuntu image from http://people.canonical.com/~platform/snappy/nuc/ubuntu-server-16.04.1-20160817-0.iso
  2. Download Win32DiskManager from http://sourceforge.net/projects/win32diskimager/files/latest/download
  3. Insert your USB flash drive and open Win32DiskImager
  4. In the Image File area, select the Ubuntu ISO image you just downloaded.
  5. In the device area, select the USB drive you are going to burn the image to, BE VERY CAREFUL, if you select the wrong drive it will be formatted.
  6. Click Write and wait for the process to finish.

Install Ubuntu Server 16.04 LTS

  1. style="margin-top:16px;margin-bottom:16px"
  2. Power up the NUC and press F10 which will enter you into the boot menu.
  3. Select the USB drive as primary boot option.
  4. When you see a prompt asking you to reboot, do so.

IoT JumpWay Intel® NUC DE3815TYKE Bugs/Issues

Adam Milton-Barker,  Intel® Software Innovator