HAND POSE RECOGNITION USING BOOSTED LOOK UP TABLES

• US 20150138078A1
• Filed: 11/18/2014
• Published: 05/21/2015
• Est. Priority Date: 11/18/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Krupka, Eyal, Vinnikov, Alon, Klein, Benjamin Eliot, Stachniak, Szymon P.
• Application Number:
  US14/546,750
• Publication Number:
  US 20150138078A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

Pose and gesture detection and classification of a human poses and gestures using a discriminative ferns ensemble classifier is provided. Sample image data in one or more channels includes a human image. A processing device operates on the sample image data using the discriminative ferns ensemble classifier. The classifier has set of classification tables and matching bit features (ferns) which are developed using a first set of training data and optimized by a weighting of the tables using an SVM linear classifier configured based on the first or a second set of pose training data. The tables allow computation of a score per pose class for the image in the sample data and the processor outputs a determination of the pose in the sample depth image data. The determination enables the manipulation of a natural user interface.

Claims

1. A gesture recognition system, comprising:

a capture device receiving image data including at least depth data; and

a processor operably coupled to the capture device including code operable to instruct the processor to classify a gesture of a human body in sample image data received by the capture device using a classifier based on a discriminative ferns ensemble.

2. The system of claim 1 wherein the classifier uses a learned model based on a first set of training data, the learned model comprises at least one optimized fern based on binary features calculated for the image data, the optimized fern being weighted based on comparison to a support vector machine classifier trained using at least one of the first set of training data and a second set of training data.

3. The system of claim 1 wherein the classifier comprises code operable to instruct a processor to:

select a patch in an image in the sample image data;

calculate multiple binary features of the image in the sample image data;

concatenate a set of binary features for the image into a word;

repeat the calculate and concatenate steps after shifting a reference point for the calculate and concatenate steps to all possible points in the image patch;

build a histogram comprising a count of a number of times each word appears in the image; and

sum all weights of cells of the histogram thereby providing a score for the fern.

4. The system of claim 3 further including wherein the processor is operable to compare each cell of the histogram to a threshold and wherein the sum of all weights comprises all weights of all cells over the threshold.

5. The system of claim 3 wherein for the classifier includes multiple lookup ferns, calculation of multiple binary features and concatenation of the set of binary features is repeated for each lookup fern, and wherein a sum the weights of the multiple lookup ferns scores is provided.

6. The system of claim 2 wherein the sample image data may include multiple channels, a first channel comprising said depth data and a second channel comprising IR data, and wherein the learned model is provided by analysis of the first set of training data including a plurality of training images, the processor operable to:

select an offsets and channels for each bit in each fern for the training image; and

assign a weight to each entry of each lookup fern for the training image.

7. The system of claim 6 wherein, for a single lookup fern, the processor is operable to create a binary vector for each training image by:

extracting binary features comparing pairs of pixels in each training image with one or more data channels based on two offsets and a channel for each feature;

evaluating the binary features by shifting a set of reference points; and

comparing the count of binary features to a threshold for every histogram bin.

8. The system of claim 6 wherein the learned model is adapted to maximize an information gain criterion, the learned model created by:

randomizing reference points by selecting at random a patch from a group of patches within the first set of training data and setting reference points to at least a subset of locations within the patch selected;

constructing binary features that compare pixels within the patch by, first, setting all binary features to return 0 constantly;

then, for every binary feature, randomizing a set of candidate binary features and for every candidate, setting a current binary feature to a candidate and consider possible combinations of thresholds for histogram bins and evaluate corresponding binary mappings for all images in the training set;

calculate the maximum information gain for binary features mappings of each candidate; and

set current binary feature to the candidate that achieved maximum information gain.

9. The system of claim 8 wherein a plurality of lookup ferns is provided and the randomizing reference points, comparing binary features and randomizing a set of candidate binary features is repeated for every lookup fern where every image in the first training set is associated with a weight which is accounted for when computing maximum information gain.

10. The system of claim 3 wherein a weight is assigned to each fern lookup table entry by using at least a first set or a second set of training data to train an SVM linear classifier with the binary features and assign weights to a resulting learned model.

13. The method of claim 1 wherein the sample image data may include multiple channels, a first channel comprising depth data and a second channel comprising IR data, and wherein the learned model is provided by analysis of the first set of training data including a plurality of training images, the method including steps:

selecting a offsets and channels for each bit in each training image; and

assigning a weight to each entry of each lookup fern for the training image.

14. The method of claim 13 wherein, for a single lookup fern, the method creates a binary vector for each training image by:

counting binary words composed of binary features obtained from each training image by comparing pairs of pixels at two offsets within the image channels;

evaluating the binary words by shifting a set of reference points; and

comparing the count of binary words to a threshold for every histogram bin.

11. A computer implemented method of classifying sample image data to determine a gesture present in the sample image data, the method comprising:

creating a discriminative ferns ensemble classifier having direct indexing to a set of classification tables, the tables developed using a learned model based on a first set of training data and optimized by a weighting of the tables using an SVM linear classifier based on at least one of one of the first set and a second set of training data;

receiving sample image data to be classified from a capture device, the capture device including one or more input channels;

analyzing the sample image data using the discriminative ferns ensemble classifier; and

outputting a determination of the gesture in the sample image data, the determination enabling a manipulation of a natural user interface.

12. The computer implemented method of claim 11 wherein the classifier performs a method of:

calculating multiple binary features of an image in the sample image data;

concatenating a set of binary features for the image into a word;

repeating said calculating and said concatenating after shifting a reference point in the sample image data to all possible points in the sample image data;

creating a histogram comprising a count of a number of times each word appears in the image; and

summing all weights in cells of the histogram thereby providing a score for the fern.

15. The method of claim 12 wherein the learned model is further created to maximize an information gain criterion by:

randomizing reference points by selecting at random a patch from a group of patches within the first set of training data and setting reference points to at least a subset of locations within the patch selected;

comparing binary features of pixels within the patch by setting all binary features to return 0 constantly;

for every binary feature, randomizing a set of candidate binary features and for every candidate, setting a current binary feature to a candidate and consider possible combinations of thresholds for histogram bins and evaluate corresponding binary mappings for all images in the training set;

calculating a maximum information gain for binary features mappings of each candidate; and

assigning the current binary feature to the candidate that achieved maximum information gain.

16. A pose detection and classification system adapted to classify human poses in sample image data, comprising:

a capture device including a first input channel and a second input channel, each channel providing sample image data;

a processing device operable on the sample image data using a discriminative ferns ensemble classifier having direct indexing to a set of classification tables, the tables developed using a first set of training data and optimized by a weighting of the tables using an SVM linear classifier configured based on a second set of training data, the processing device a outputting a determination of the pose in the sample image data, the determination enabling a manipulation of a natural user interface.

17. The system of claim 16 wherein the ensemble classifier comprises code operable to instruct a processor to:

select a patch in an image in the sample image data;

calculate multiple binary features of the image in the sample image data;

concatenate a set of binary features for the image into a word;

repeat the calculate and concatenate steps after shifting a reference point for the calculate and concatenate steps to all possible points in the image;

build a histogram comprising a count of a number of times each word appears in the image; and

sum all weights of cells of the histogram thereby providing a score for the fern.

18. The system of claim 17 further including wherein the processor is operable to compare each cell of the histogram to a threshold and wherein the sum of all weights comprises a sum of all weights of all cells over the threshold.

19. The system of claim 18 wherein for the classifier includes multiple lookup ferns, calculation of multiple binary features and concatenation of the set of binary features is repeated for each lookup fern, and wherein a sum of weights of multiple lookup ferns scores is provided.

20. The system of claim 19 wherein the classification tables are adapted to maximize an information gain criterion, the classification tables created by:

randomizing reference points by selecting at random a patch from a group of patches within the first set of training data and setting reference points to at least a subset of locations within the patch selected;

learning binary features based on pixel comparisons within the patch by first setting all binary features to return zero constantly;

for every binary feature, randomizing a set of candidate binary features and for every candidate, setting a current binary feature to a candidate and consider possible combinations of thresholds for histogram bins and evaluate corresponding binary mappings for all images in the training set;

calculate the maximum information gain for binary features mappings of each candidate; and

set current binary feature to the candidate that achieved maximum information gain.