HAND GESTURE RECOGNITION FOR VIRTUAL REALITY AND AUGMENTED REALITY DEVICES

• US 20180189556A1
• Filed: 01/03/2017
• Published: 07/05/2018
• Est. Priority Date: 01/03/2017
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Granted Patent: US 10,354,129 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system for hand gesture recognition includes a display, camera, memory, and processor. When the processor is to execute instructions, the processor is to estimate one or more motion vectors of an object using a pair of consecutive frames and estimate an average motion vector of the object. The processor may also estimate an average motion vector of the object, determine a first histogram of optical flow (HOOF) based on the one or more motion vectors and the average motion vector, determine depth values based on motion vectors from the first HOOF, and determine a second histogram of optical flow (HOOF) based on the depth values. The processor is also to obtain a descriptor based on histogram values from a histogram of optical flow (HOOF) of the one or more motion vectors, a shape descriptor, and the average motion vector and classify the descriptor as a gesture.

Claims

1. A system for hand gesture recognition, comprising:

a display;

a camera;

a memory that is to store instructions and that is communicatively coupled to the camera and the display; and

a processor communicatively coupled to the camera, the display, and the memory, wherein when the processor is to execute the instructions, the processor is to;

estimate one or more motion vectors of an object using a pair of consecutive frames;

estimate an average motion vector of the object;

determine a first histogram of optical flow (HOOF) based on the one or more motion vectors and the average motion vector;

determine depth values based on the first HOOF;

determine a second histogram of optical flow (HOOF) based on the depth values;

calculate a descriptor using the first HOOF, the second HOOF, a shape descriptor, and an average motion vector; and

classify the descriptor as a gesture.

2. The system of claim 1, comprising:

extracting a mask of the object using a depth image; and

applying the optical flow to the mask of the object to obtain the one or more motion vectors.

3. The system of claim 1, wherein the histogram of the optical flow (HOOF) of the one or more motion vectors is built using a Lucas-Kanade optical flow algorithm.

4. The system of claim 1, wherein the histogram values and the average motion vectors are concatenated for a plurality of frames to obtain the descriptor.

5. The system of claim 1, wherein each descriptor includes histogram values and the average motion vector for K frames, wherein the value of K is dependent on the type of gesture.

6. The system of claim 1, wherein a machine learning algorithm is used to classify the descriptor as a gesture.

7. The system of claim 1, wherein the one or more motion vectors is calculated via an optical flow algorithm of a mask applied an infrared image, a red, green, blue (RGB) image, or a grayscale image.

8. The system of claim 1, wherein the average motion vector is estimated using good feature points are determined by corner tracking.

9. The system of claim 1, wherein the object is a hand.

10. The system of claim 1, wherein the gesture is used to control a cursor.

11. A method, comprising:

extracting a hand mask using depth data;

estimating a first plurality of motion vectors based via an optical flow applied to the hand mask;

estimating an average motion vector from the optical flow;

estimating a second plurality of motion vectors via depth values derived from the first plurality of motion vectors;

generating a descriptor based on a histogram of optical flow applied to the first plurality of motion vectors, the second plurality of motion vectors, the hand mask, and the average motion vector; and

classifying the descriptor as a gesture.

12. The method of claim 11, wherein the histogram of optical flow is calculated for a plurality of cells of a frame, and a value is calculated for each bin of the histogram of optical flow by average values of the bin.

13. The method of claim 12, wherein the average bin values and the average motion vector are concatenated for a plurality of frames to generate the descriptor.

14. The method of claim 11, comprising:

obtaining the plurality of motion vectors for a plurality of good feature points in a sequence of frames;

selecting the motion vectors with a lowest length; and

averaging the motion vectors with the lowest length to obtain the average motion vector.

15. The method of claim 11, wherein the descriptor is invariant to rotation and hand type.

16. The method of claim 11, wherein a movement corresponding to the gesture is applied to a cursor rendered on a display.

17. An apparatus for hand gesture recognition, comprising:

an image capture mechanism to obtain a sequence of frames;

an estimator to estimate a first plurality of motion vectors, a second plurality of motion vectors, and an average motion vector for each frame of the sequence of frames;

a generator to generate a descriptor based on a first histogram of the first plurality of motion vectors, a second histogram based on the second plurality of motion vectors, a shape descriptor, and the average motion vector for the sequence of frames; and

a classifier to classify the descriptor as a gesture.

18. The apparatus of claim 17, wherein the histogram is a histogram of optical flow, and the one or more motion vectors is projected onto the histogram of optical flow.

19. The apparatus of claim 17, wherein a frame is divided into a plurality of cells, and a histogram of optical flow is calculated for each cell, and in each bin the histogram of optical flow is averaged to obtain the single frame feature vector that comprises histogram values.

20. The apparatus of claim 17, wherein histogram values and the average motion vector for each frame of the sequence of frames are concatenated to generate the descriptor.

21. The apparatus of claim 17, wherein each descriptor includes a set of histogram values and the average motion vector for K frames, wherein the value of K is dependent on the type of gesture.

22. A tangible, non-transitory, computer-readable medium comprising instructions that, when executed by a processor, direct the processor to:

extract a hand mask using depth data;

estimate a first plurality of motion vectors based via an optical flow applied to the hand mask;

estimate an average motion vector from the optical flow;

estimate a second plurality of motion vectors via depth values derived from the first plurality of motion vectors;

generate a descriptor based on a histogram of optical flow applied to the hand mask and the average motion vector; and

classify the descriptor as a gesture.

23. The computer readable medium of claim 22, wherein each descriptor includes a concatenated plurality of histogram values and the average motion vector for K frames, wherein the value of K is dependent on a type of the gesture.

24. The computer readable medium of claim 22, wherein a machine learning algorithm is used to classify the descriptor as a gesture.

25. The computer readable medium of claim 22, wherein the hand mask is applied to an infrared image, a red, green, blue (RGB) image, or a grayscale image to estimate the plurality of motion vectors.