Image-based real-time gesture recognition

• US 9,128,528 B2
• Filed: 06/22/2012
• Issued: 09/08/2015
• Est. Priority Date: 06/22/2012
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Tian, Dihong
• Primary Examiner(s): Patel, Hemant
• Application Number:
  US13/530,698
• Publication Number:
  US 20130342636A1
• Time in Patent Office: 1,173 Days
• Field of Search: 348 1401- 1416, 370259-271, 370351-357, 709201-207, 709217-248
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 7/12 Systems in which the television signal is transmitted via one channel or a plurality of parallel channels, the bandwidth of each channel being less than the bandwidth of the television signal H04N7/24 takes precedence
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Techniques are provided for image-based real-time gesture recognition. Video data of a person is obtained. Pixels are classified in the video stream at a given time instance during a time period as a foreground or a background pixel. A data entry is generated comprising data indicating foreground history values for each of a plurality of time instances of the video stream and data indicating a time period value. When the classifying indicates that a first pixel is a foreground pixel, the data structure associated with the first pixel is evaluated to determine whether or not to update a foreground history value associated with the first pixel at the given time instance. A motion gradient vector is generated for the video stream based on the foreground history value associated with the first pixel and foreground history values associated with other pixels.

Claims

1. A method comprising:

obtaining video data comprising a video stream of a person;

classifying a first pixel in the video stream at an initial time instance of a time period as one of a foreground pixel or a background pixel;

initializing a foreground history value associated with the first pixel to zero for the initial time instance;

reclassifying the first pixel at each time instance subsequent to the beginning time instance in the time period as one of a foreground pixel or a background pixel;

when reclassification indicates that the first pixel at a first time instance is a foreground pixel, setting the foreground history value associated with the first pixel to a non-zero value related to the first time instance;

maintaining the foreground history value associated with the first pixel to the non-zero value related to the first time instance for a pre-determined time period;

resetting the foreground history value to zero at a second time instance after the pre-determined time period expires; and

generating a motion gradient vector for the video stream based on the foreground history value associated with the first pixel and foreground history values associated with other pixels over the time period.

2. The method of claim 1, further comprising dividing the video data into one or more video data regions and performing the classifying, generating the data entry and evaluating operations on pixels in each of the video data regions.

3. The method of claim 1, further comprising categorizing each pixel that is designated as a foreground pixel into one or more foreground groups.

4. The method of claim 3, further comprising:

generating an expected gradient vector for each pixel based on the categorizing; and

comparing the expected gradient vector to the motion gradient vector to obtain a movement determination for the foreground group.

5. The method of claim 1, further comprising:

interpreting the motion gradient vector as being associated with a predetermined gesture intended by the person; and

executing an electronic operation associated with the predetermined gesture intended by the person.

6. The method of claim 1, wherein generating the motion gradient vector comprises generating a grayscale image of foreground pixels in the time period.

7. The method of claim 6, wherein generating the motion gradient vector comprises convolving the grayscale image with a predetermined edge-detection kernel.

22. The method of claim 5, further comprising:

evaluating the motion gradient vector with respect to each of a plurality of predetermined gestures, to produce a confidence value for each of the plurality of predetermined gestures; and

selecting one of the plurality of predetermined gestures as the predetermined gesture intended by the person, wherein the predetermined gesture intended by the person is associated with a highest confidence value from among the plurality of predetermined gestures.

8. One or more non-transitory computer readable storage media encoded with software comprising computer executable instructions and when the software is executed, the software is operable to:

obtain video data comprising a video stream of a person;

classify a first pixel in the video stream at an initial time instance of a time period as one of a foreground pixel or a background pixel;

initialize a foreground history value associated with the first pixel to zero for the initial time instance;

reclassify the first pixel at each time instance subsequent to the beginning time instance in the time period as one of a foreground pixel or a background pixel;

when reclassification indicates that the first pixel at a first time instance is a foreground pixel, set the foreground history value associated with the first pixel to a non-zero value related to the first time instance;

maintain the foreground history value associated with the first pixel to the non-zero value related to the first time instance for a pre-determined time period;

reset the foreground history value to zero at a second time instance after the pre-determined time period expires; and

generate a motion gradient vector for the video stream based on the foreground history value associated with the first pixel and foreground history values associated with other pixels over the time period.

9. The computer readable storage media of claim 8, further comprising instructions operable to divide the video data into one or more video data regions and performing the classify operation, generate the data entry operation and evaluate operations on pixels in each of the video data regions.

10. The computer readable storage media of claim 8, further comprising instructions operable to categorize each pixel that is designated as a foreground pixel into one or more foreground groups.

11. The computer readable storage media of claim 10, further comprising instructions operable to:

generate an expected gradient vector for each pixel based on the categorize operation; and

compare the expected gradient vector to the motion gradient vector to obtain a movement determination for the foreground group.

12. The computer readable storage media of claim 8, further comprising instructions operable to:

interpret the motion gradient vector as being associated with a predetermined gesture intended by the person; and

execute an electronic operation associated with the predetermined gesture intended by the person.

13. The computer readable storage media of claim 8, wherein the instructions operable to generate the motion gradient vector comprise instructions operable to generate a grayscale image with a predetermined edge-detection kernel.

14. The computer readable storage media of claim 13, wherein the instructions operable to generate the motion gradient vector comprise instructions operable to convolve the grayscale image with a predetermined edge-detection kernel.

23. The computer readable storage media of claim 12, further comprising instructions operable to:

evaluate the motion gradient vector with respect to each of a plurality of predetermined gestures, to produce a confidence value for each of the plurality of predetermined gestures; and

select one of the plurality of predetermined gestures as the predetermined gesture intended by the person, wherein the predetermined gesture intended by the person is associated with a highest confidence value from among the plurality of predetermined gestures.

15. An apparatus comprising:

a network interface unit;

a memory; and

a processor coupled to the network interface unit and the memory and configured to;

obtain video data comprising a video stream of a person;

classify a first pixel in the video stream at an initial time instance of a time period as one of a foreground pixel or a background pixel;

initialize a foreground history value associated with the first pixel to zero for the initial time instance;

reclassify the first pixel at each time instance subsequent to the beginning time instance in the time period as one of a foreground pixel or a background pixel;

when reclassification indicates that the first pixel at a first time instance is a foreground pixel, set the foreground history value associated with the first pixel to a non-zero value related to the first time instance;

maintain the foreground history value associated with the first pixel to the non-zero value related to the first time instance for a pre-determined time period;

reset the foreground history value to zero at a second time instance after the pre-determined time period expires; and

generate a motion gradient vector for the video stream based on the foreground history value associated with the first pixel and foreground history values associated with other pixels over the time period.

16. The apparatus of claim 15, wherein the processor is further configured to divide the video data into one or more video data regions and to perform the classify, generate the data entry and evaluate operations on pixels in each of the video data regions.

17. The apparatus of claim 15, wherein the processor is further configured to categorize each pixel that is designated as a foreground pixel into one or more foreground groups.

18. The apparatus of claim 17, wherein the processor is further configured to:

generate an expected gradient vector for each pixel based on the categorize operation; and

compare the expected gradient vector to the motion gradient vector to obtain a movement determination for the foreground group.

19. The apparatus of claim 15, wherein the processor is further configured to:

interpret the motion gradient vector as being associated with a predetermined gesture intended by the person; and

execute an electronic operation associated with the predetermined gesture intended by the person.

20. The apparatus of claim 15, wherein the processor is further configured to generate a grayscale image of foreground pixels in the time period.

21. The apparatus of claim 20, wherein the processor is further configured to convolve the grayscale image with a predetermined edge-detection kernel.

24. The apparatus of claim 19, wherein the processor is further configured to:

evaluate the motion gradient vector with respect to each of a plurality of predetermined gestures, to produce a confidence value for each of the plurality of predetermined gestures; and

select one of the plurality of predetermined gestures as the predetermined gesture intended by the person, wherein the predetermined gesture intended by the person is associated with a highest confidence value from among the plurality of predetermined gestures.