METHOD AND SYSTEM FOR GESTURE-BASED HUMAN-MACHINE INTERACTION AND COMPUTER-READABLE MEDIUM THEREOF

• US 20130162532A1
• Filed: 08/16/2011
• Published: 06/27/2013
• Est. Priority Date: 09/17/2010
• Status: Active Grant

• Current Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Inventors: Cheng, Tong, Yue, Shuai
• Granted Patent: US 8,963,836 B2
• Time in Patent Office: Days
• US Class Current: 345/157
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A method and system for gesture-based human-machine interaction and computer-readable medium are provided. The system includes a capturing module, a positioning module, and a transforming module. The method includes the steps of: capturing images from a user'"'"'s video streams, positioning coordinates of three or more predetermined color blocks in the foreground, simulating movements of a mouse according to the coordinates of the first color block, and simulating click actions of the mouse according to the coordinates of the other color blocks. The embodiments according to the current disclosure position coordinates of a plurality of color blocks through processing the captured user'"'"'s video streams, and simulate mouse actions according to the coordinates of the color blocks. Processing apparatuses like computers may be extended to facilitate gesture-based human-machine interactions through a very simple way, and a touch-sensitive interaction effect can be simulated, without the presence of a touch screen.

Claims

1. A method for gesture-based human-machine interaction, comprising:

capturing images from a user'"'"'s video streams;

positioning coordinates of three or more predetermined color blocks in the foreground; and

simulating movements of a mouse according to the coordinates of the first color block, and simulating click actions of the mouse according to the coordinates of the other color blocks.

2. The method for gesture-based human-machine interaction according to claim 1, wherein the step of capturing user'"'"'s video stream and capturing images from user'"'"'s video streams comprises,

capturing user'"'"'s video stream by using a video camera, and capturing continuous images from the video stream, or capturing discrete images from the video stream in a predetermined interval.

3. The method for gesture-based human-machine interaction according to claim 2, wherein the step of positioning coordinates of three or more predetermined color blocks in the foreground comprises,

setting up a background model, for generating a mask picture, and removing the background of the images by the mask picture;

obtaining predetermined a first color block template, a second color block template and a third color block template, and generating a histogram corresponding to each color block according to the images;

calculating a probability distribution diagram of each color block according to the corresponding histogram; and

tracing a centroid of each color block in the probability distribution diagram, to determine the coordinates of the three color blocks.

4. The method for gesture-based human-machine interaction according to claim 3, wherein the step of simulating click actions of the mouse according to the coordinates of the other color blocks comprises,

determining a straight line distance between the coordinates of the second color block and the third color block, and

determining if the straight line distance is less than a predetermined threshold value such that to simulate a press action of a left key of the mouse if the straight line distance is less than the predetermined threshold value, or to simulate a release action of the left key of the mouse if the straight line distance is not less than the predetermined threshold value.

5. The method for gesture-based human-machine interaction according to claim 4, wherein the step of obtaining predetermined three color block templates comprises,

wearing three sleeves on different fingers with different colors, and storing the colors of the sleeves as templates for the color blocks.

6. The method for gesture-based human-machine interaction according to claim 3, further comprising the following steps after the step of calculating the probability distribution diagram of each color block according to the corresponding histogram:

removing noise of the probability distribution diagrams by using a noise erosion operation; and

applying Gaussian smoothing process on the probability distribution diagrams, and executing threshold segmentation on the Gaussian smoothed probability distribution diagrams.

7. A system for gesture-based human-machine interaction, comprising:

a capturing module configured to capture images from a user'"'"'s video streams;

a positioning module configured to position coordinates of three or more predetermined color blocks in the foreground; and

a transforming module configured to simulate movements of a mouse according to the coordinates of the first color block, and simulating click actions of the mouse according to the coordinates of the other color blocks.

8. The system for gesture-based human-machine interaction according to claim 7, wherein the capturing module captures the user'"'"'s video stream through a video camera, and captures continuous images from the video stream, or capturing discrete images from the video stream in a predetermined interval.

9. The system for gesture-based human-machine interaction according to claim 8, wherein the positioning module comprises,

a background segmentation unit configured to set up a background model for generating a mask picture, and removing the background of the images by the mask picture;

a histogram unit configured to obtain a first predetermined color block template, a second predetermined color block template and a third predetermined color block template, and generating a histogram corresponding to each color block according to the images;

a probability distribution diagram unit configured to calculate a probability distribution diagram of each color block according to the corresponding histogram; and

a trace unit configured to trace a centroid of each color block in the probability distribution diagram, to determine the coordinates of the three color blocks.

10. The system for gesture-based human-machine interaction according to claim 9, wherein the transforming module is configured to determine the straight line distance between the coordinates of the second color block and the third color block, and to determine if the straight line distance is less than a predetermined threshold value;

to simulate a press action of a left key of the mouse if the straight line distance is less than the predetermined threshold value;

to simulate a release action of the left key of the mouse if the straight line distance is not less than the predetermined threshold value.

11. The system according to claim 10, wherein the system further comprises, three sleeves with different colors wore on different fingers, wherein the colors of the sleeves are stored as templates for the color blocks.

12. The system according to claim 9, wherein the positioning module further comprises an optimization unit, wherein the optimization unit comprises a noise removing subunit and a smoothing subunit, wherein the noise removing subunit is used to remove noise by applying noise erosion operation on the probability distribution diagrams, and the smoothing subunit is used to apply a Gaussian smoothing process on the probability distribution diagrams, and executing threshold segmentation on the Gaussian smoothed diagrams.

13. At least one non-transitory computer-readable medium having computer-executive instructions stored thereon that, when executed by a processor, cause a computer to implement a gesture-based human-machine interaction method, the method comprising:

capturing images from user'"'"'s video streams;

positioning coordinates of three or more predetermined color blocks in the foreground; and

simulating movements of a mouse according to the coordinates of the first color block, and simulating click actions of the mouse according to the coordinates of the other color blocks.

14. The non-transitory computer-readable medium according to claim 13, wherein the step of capturing user'"'"'s video stream and capturing images from user'"'"'s video streams comprises,

capturing a user'"'"'s video stream by using a video camera, and capturing continuous images from the video stream, or capturing discrete images from the video stream in a predetermined interval.

15. The non-transitory computer-readable medium according to claim 14, wherein the step of positioning coordinates of three or more predetermined color blocks in the foreground comprises,

setting up a background model, for generating a mask picture, and removing the background of the images by the mask picture;

obtaining a first predetermined color block template, a second predetermined color block template and a third predetermined color block template, and generating a histogram corresponding to each color block according to the images;

calculating a probability distribution diagram of each color block according to the corresponding histogram;

tracing a centroid of each color block in the probability distribution diagram to determine the coordinates of the three color blocks.

16. The non-transitory computer-readable medium according to claim 15, wherein the step of simulating click actions of the mouse according to the coordinates of the other color blocks comprises,

determining the straight line distance between the coordinates of the second color block and the third color block, and

determining if the straight line distance is less than a predetermined threshold value such that to simulate a press action of a left key of the mouse if the straight line distance is less than the predetermined threshold value, or simulate a release action of the left key of the mouse if the straight line distance is not less than the predetermined threshold value.

17. The non-transitory computer-readable medium according to claim 16, wherein the step of obtaining predetermined three color block templates comprises,

wearing three sleeves on different fingers with different colors, and storing the colors of the sleeves as templates for the color blocks.

18. The non-transitory computer-readable medium according to claim 15, wherein the following steps are comprised after the step of calculating the probability distribution diagram of each color block according to the corresponding histogram,

removing noise from the probability distribution diagrams by applying a noise erosion operation;

applying a Gaussian smoothing on the probability distribution diagrams, and executing threshold segmentation on the Gaussian smoothed probability distribution diagrams.