Image-Based Real-Time Gesture Recognition

• US 20130342636A1
• Filed: 06/22/2012
• Published: 12/26/2013
• Est. Priority Date: 06/22/2012
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Tian, Dihong
• Granted Patent: US 9,128,528 B2
• Time in Patent Office: Days
• US Class Current: 348/14.08
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 7/12 Systems in which the television signal is transmitted via one channel or a plurality of parallel channels, the bandwidth of each channel being less than the bandwidth of the television signal H04N7/24 takes precedence
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Techniques are provided for image-based real-time gesture recognition. Video data of a person is obtained. Pixels are classified in the video stream at a given time instance during a time period as a foreground or a background pixel. A data entry is generated comprising data indicating foreground history values for each of a plurality of time instances of the video stream and data indicating a time period value. When the classifying indicates that a first pixel is a foreground pixel, the data structure associated with the first pixel is evaluated to determine whether or not to update a foreground history value associated with the first pixel at the given time instance. A motion gradient vector is generated for the video stream based on the foreground history value associated with the first pixel and foreground history values associated with other pixels.

Claims

1. A method comprising:

obtaining video data comprising a video stream of a person;

classifying pixels in the video stream at a given time instance during a time period as one of a foreground pixel and a background pixel;

generating a data entry in a data structure corresponding to each pixel, wherein the data structure comprises data indicating foreground history values for each of a plurality of time instances of the video stream and data indicating a time period value;

when classifying indicates that a first pixel of the video stream is a foreground pixel, evaluating the data structure associated with the first pixel to determine whether or not to update the data representing a foreground history value associated with the first pixel at the given time instance; and

generating a motion gradient vector for the video stream based on the foreground history value associated with the first pixel and foreground history values associated with other pixels over the time period.

2. The method of claim 1, further comprising updating the data representing the foreground history value associated with the first pixel when evaluating determines that a foreground history value associated with the first pixel at an immediate prior time instance indicates that the first pixel was not classified as a foreground pixel at the immediate prior time instance.

3. The method of claim 1, further comprising dividing the video data into one or more video data regions and performing the classifying, generating the data entry and evaluating operations on pixels in each of the video data regions.

4. The method of claim 1, further comprising categorizing each pixel that is designated as a foreground pixel into one or more foreground groups.

5. The method of claim 4, further comprising:

generating an expected gradient vector for each pixel based on the categorizing; and

comparing the expected gradient vector to the motion gradient vector to obtain a movement determination for the foreground group.

6. The method of claim 1, further comprising:

interpreting the motion gradient vector as being associated with a predetermined gesture intended by the person; and

executing an electronic operation associated with the gesture.

7. The method of claim 1, wherein generating the motion gradient vector comprises generating a grayscale image of foreground pixels in the time period.

8. The method of claim 7, wherein generating the motion gradient vector comprises convolving the grayscale image with a predetermined edge-detection kernel.

9. One or more computer readable storage media encoded with software comprising computer executable instructions and when the software is executed operable to:

obtain video data comprising a video stream of a person;

classify pixels in the video stream at a given time instance during a time period as one of a foreground pixel and a background pixel;

generate a data entry in a data structure corresponding to each pixel, wherein the data structure comprises data indicating foreground history values for each of a plurality of time instances of the video stream and data indicating a time period value;

when a first pixel of the video stream is a foreground pixel, evaluate the data structure associated with the first pixel to determine whether or not to update the data representing a foreground history value associated with the first pixel at the given time instance; and

generate a motion gradient vector for the video stream based on the foreground history value associated with the first pixel and foreground history values associated with other pixels over the time period.

10. The computer readable storage media of claim 9, further comprising instructions operable to update the data representing the foreground history value associated with the first pixel when the evaluation operation determines that a foreground history value associated with the first pixel at an immediate prior time instance indicates that the first pixel was not classified as a foreground pixel at the immediate prior time instance.

11. The computer readable storage media of claim 9, further comprising instructions operable to divide the video data into one or more video data regions and performing the classify operation, generate the data entry operation and evaluate operations on pixels in each of the video data regions.

12. The computer readable storage media of claim 9, further comprising instructions operable to categorize each pixel that is designated as a foreground pixel into one or more foreground groups.

13. The computer readable storage media of claim 12, further comprising instructions operable to:

generate an expected gradient vector for each pixel based on the categorize operation; and

compare the expected gradient vector to the motion gradient vector to obtain a movement determination for the foreground group.

14. The computer readable storage media of claim 9, further comprising instructions operable to:

interpret the motion gradient vector as being associated with a predetermined gesture intended by the person; and

execute an electronic operation associated with the gesture.

15. The computer readable storage media of claim 9, wherein the instructions operable to generate the motion gradient vector comprise instructions operable to generate a grayscale image with a predetermined edge-detection kernel.

16. The computer readable storage media of claim 15, wherein the instructions operable to generate the motion gradient vector comprise instructions operable to convolve the grayscale image with a predetermined edge-detection kernel.

17. An apparatus comprising:

a network interface unit;

a memory; and

a processor coupled to the network interface unit and the memory and configured to;

obtain video data comprising a video stream of a person;

classify pixels in the video stream at a given time instance during a time period as one of a foreground pixel and a background pixel;

generate a data entry in a data structure corresponding to each pixel, wherein the data structure comprises data indicating foreground history values for each of a plurality of time instances of the video stream and data indicating a time period value;

when a first pixel of the video stream is a foreground pixel, evaluate the data structure associated with the first pixel to determine whether or not to update the data representing a foreground history value associated with the first pixel at the given time instance; and

generate a motion gradient vector for the video stream based on the foreground history value associated with the first pixel and foreground history values associated with other pixels over the time period.

18. The apparatus of claim 17, wherein the processor is further configured to update the data representing the foreground history value associated with the first pixel when it is determined that a foreground history value associated with the first pixel at an immediate prior time instance indicates that the first pixel was not classified as a foreground pixel at the immediate prior time instance.

19. The apparatus of claim 17, wherein the processor is further configured to divide the video data into one or more video data regions and to perform the classify, generate the data entry and evaluate operations on pixels in each of the video data regions.

20. The apparatus of claim 17, wherein the processor is further configured to categorize each pixel that is designated as a foreground pixel into one or more foreground groups.

21. The apparatus of claim 20, wherein the processor is further configured to:

generate an expected gradient vector for each pixel based on the categorize operation; and

compare the expected gradient vector to the motion gradient vector to obtain a movement determination for the foreground group.

22. The apparatus of claim 17, wherein the processor is further configured to:

interpret the motion gradient vector as being associated with a predetermined gesture intended by the person; and

execute an electronic operation associated with the gesture.

23. The apparatus of claim 17, wherein the processor is further configured to generate a grayscale image of foreground pixels in the time period.

24. The apparatus of claim 17, wherein the processor is further configured to convolve the grayscale image with a predetermined edge-detection kernel.