Can I use the GRT with a Kinect?

Yes. The GRT gesture-recognition modules can accept any of sensor input, as long as you can get your sensor data into a c++ double vector format. You can therefore get the GRT to work with the skeleton data from a Kinect sensor by taking the x, y, and z points from each skeleton joint of a tracked user (or just a few key joints such as the head and hands) and adding the 3D coordinates from each joint to a c++ vector. For example, if you wanted to use the GRT to recognize left hand and right hand gestures then you would do the following somewhere in your code:

//Create a new double vector to hold the left and right hand skeleton data
vector< double > inputVector(6);

//Add the joint data from the Kinect to the inputVector
inputVector[0] = kinectLeftHandData[0];
inputVector[1] = kinectLeftHandData[1];
inputVector[2] = kinectLeftHandData[2];
inputVector[3] = kinectRightHandData[0];
inputVector[4] = kinectRightHandData[1];
inputVector[5] = kinectRightHandData[2];

//You can now input this data into a GestureRecognitionPipeline and use this to recognize your gestures (assuming you have trained the pipeline already)
pipeline.predict( inputVector );

The example code above assumes that you have two arrays containing the Kinect joint data (these arrays have nothing to do with the GRT, this is something you need to write based on whatever API you are using to get the joint data from the Kinect).

Can I use the GRT with a Leap?


The GRT gesture-recognition modules can accept any of sensor input, as long as you can get your sensor data into a c++ double vector format. You can therefore get the GRT to work with the Leap by taking the 3D position and rotation values that the Leap SDK gives you and concatenating these values into one c++ vector which you can then use as input to the GRT.

Note that, because of occlusion of key joints by the hand and other fingers etc., you should choose carefully what values you use as input to the GRT. For example, if you wanted to use the GRT to recognize if the user was making a right hand swipe gesture, then it probably makes sense only to use the 3D centroid value for the right hand as input (and ignore any finger values from the right hand and also all values from the left hand). One of the main disadvantages of the data that comes from the Leap SDK (at least in the early versions of the SDK) is that the data often requires extensive filtering from sophisticated tracking algorithms (such as Kalman filters or particle filters) to mitigate occlusions and erroneous sensor estimates. Without such filtering, it makes recognizing certain gestures difficult as there are frequently samples when not all of the data for the key joints you need are available.

How do I record some training data to train a classifier

Before you can use any of the GRT algorithms to recognize your real-time gestures, you first need to train a classification model. To train a classification model, you need to record some examples of the gestures you want the classifier to recognize and then use this training data to train the classification model.

The GRT has a large number of utilities and data structures to assist you to record, label, manage, save and load training data. For example, if you are using an ANBC, KNN, GMM or SVM classifier then you should record your training data using the LabelledClassificationData. The LabelledClassificationData is the default data structure for labelled classification data, which simply means that each data sample (which is a c++ double vector) has a corresponding label (which is an unsigned integer) that represents the class label (i.e. gesture label) of that specific data sample.

If you have already recorded some labelled data, then you can use the LabelledClassificationData class to load CSV data directly from a file, allowing you to record, label and edit your training data in another program (such as Excel or Matlab). In the majority of cases however, you will want to record and label your training data on the fly. You can easily do this using the LabelledClassificationData. The code below demonstrates how to do this.

//STEP 1 - Create a new instance of the LabelledClassificationData and set the dimensionality of your data
LabelledClassificationData trainingData;

//Set the dimensionality of the data, in this example our data has 3 dimensions
trainingData.setNumDimensions( 3 );

//STEP 2 - Get the realtime data from your sensor and label it with the corresponding gesture it belongs to
//NOTE: you would do this many times for every gesture not just once
UINT gestureLabel = 1;
vector< double > sample(3);
sample[0] = //....Data from sensor
sample[1] = //....Data from sensor
sample[2] = //....Data from sensor

//Add the sample to the training data
trainingData.addSample( gestureLabel, sample );

//Step 3 - After recording your training data you can then save it to a file
bool saveResult = trainingData.saveDatasetToFile( "TrainingData.txt" );

//This can then be loaded later
bool loadResult = trainingData.loadDatasetFromFile( "TrainingData.txt" );

where UINT is a GRT type representing an unsigned int.

You can find a complete example of how to record, label, manage, save and load training data on the LabelledClassificationData reference page. The reference page also contains a link to the full documentation for the LabelledClassificationData class.

I've developed my own custom feature extraction algorithm and want to use it with the GRT classification algorithms, is this possible?

Yes. One of the main advantages of the GRT design is that you can write your own custom feature-extraction algorithms and easily add them to the GRT framework. The GestureRecognitionPipeline can then call your custom feature-extraction algorithm just as it would call any of the core GRT modules. To do this all you need to do is write a small wrapper class that inherits from the GRT::FeatureExtraction base class. You can then add your custom feature-extraction code to this wrapper class and ta-da, the GRT can now call your algorithm. It is also possible to add your own pre-processing, post-processing, classifier, and regression modules simply by writing a wrapper class for your code that inherits from one of the main GRT base classes.