Method and system for gesture-based human-machine interaction and computer-readable medium thereof

• US 8,963,836 B2
• Filed: 08/16/2011
• Issued: 02/24/2015
• Est. Priority Date: 09/17/2010
• Status: Active Grant

• Current Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Shenzhen Company Limited (Tencent Holdings Limited)
• Inventors: Cheng, Tong, Yue, Shuai
• Primary Examiner(s): Chow, Van
• Application Number:
  US13/697,518
• Publication Number:
  US 20130162532A1
• Time in Patent Office: 1,288 Days
• Field of Search: 345/156
• US Class Current: 345/157
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A method and system for gesture-based human-machine interaction and computer-readable medium are provided. The system includes a capturing module, a positioning module, and a transforming module. The method includes the steps of: capturing images from a user'"'"'s video streams, positioning coordinates of three or more predetermined color blocks in the foreground, simulating movements of a mouse according to the coordinates of the first color block, and simulating click actions of the mouse according to the coordinates of the other color blocks. The embodiments according to the current disclosure position coordinates of a plurality of color blocks through processing the captured user'"'"'s video streams, and simulate mouse actions according to the coordinates of the color blocks. Processing apparatuses like computers may be extended to facilitate gesture-based human-machine interactions through a very simple way, and a touch-sensitive interaction effect can be simulated, without the presence of a touch screen.

Claims

1. A method for gesture-based human-machine interaction, comprising:

capturing images from a video stream of a user, wherein each of the images comprises a foreground and a background, and the foreground comprises three or more predetermined color blocks, wherein the three or more predetermined color blocks comprises at least one color block corresponding to one hand of the user, and at least two color blocks corresponding to the other hand of the user;

positioning coordinates of the three or more predetermined color blocks in the foreground of the images; and

simulating movements of a mouse according to the coordinates of the at least one color block corresponding to the one hand of the user, and simulating click actions of the mouse according to the coordinates of the at least two color blocks corresponding to the other hand of the user.

2. The method for gesture-based human-machine interaction according to claim 1, wherein the step of capturing images comprises:

capturing the video stream using a video camera, and capturing continuous images from the video stream, or capturing discrete images from the video stream in a predetermined interval.

3. The method for gesture-based human-machine interaction according to claim 2, wherein the step of positioning coordinates comprises:

setting up a background model for generating a mask picture, and removing the background of the images by the mask picture;

obtaining predetermined templates of a first color block, a second color block and a third color block of the three or more predetermined color blocks, wherein the first color block corresponds to one finger of the one hand of the user, and the second and third color blocks respectively correspond to two fingers of the other hand of the user;

generating a histogram corresponding to each of the first, second and third color blocks according to the images;

calculating a probability distribution diagram of each of the first, second and third color blocks according to the corresponding histogram; and

tracing a centroid of each of the first, second and third color blocks in the probability distribution diagram, to determine the coordinates of the first, second and third color blocks.

4. The method for gesture-based human-machine interaction according to claim 3, wherein the step of simulating click actions of the mouse according to the coordinates of the at least two color blocks corresponding to the other hand of the user comprises:

determining a straight line distance between the coordinates of the second color block and the third color block,

determining if the straight line distance is less than a predetermined threshold value;

simulating a press action of a left key of the mouse if the straight line distance is less than the predetermined threshold value; and

simulating a release action of the left key of the mouse if the straight line distance is not less than the predetermined threshold value.

5. The method for gesture-based human-machine interaction according to claim 4, wherein the step of obtaining predetermined templates comprises:

providing three sleeves with three different colors on three different fingers of the user, and

storing the colors of the three sleeves correspondingly as the predetermined templates for the first, second and third color blocks.

6. The method for gesture-based human-machine interaction according to claim 3, further comprising the following steps after the step of calculating the probability distribution diagram:

removing noise of the probability distribution diagrams by using a noise erosion operation; and

applying Gaussian smoothing process on the probability distribution diagrams, and executing threshold segmentation on the Gaussian smoothed probability distribution diagrams.

7. A system for gesture-based human-machine interaction, comprising:

a capturing module configured to capture images from a video stream of a user, wherein each of the images comprises a foreground and a background, and the foreground comprises three or more predetermined color blocks, wherein the three or more predetermined color blocks comprises at least one color block corresponding to one hand of the user, and at least two color blocks corresponding to the other hand of the user;

a positioning module configured to position coordinates of the three or more predetermined color blocks in the foreground of the images; and

a transforming module configured to simulate movements of a mouse according to the coordinates of the at least one color block corresponding to the one hand of the user, and to simulate click actions of the mouse according to the coordinates of the at least two color blocks corresponding to the other hand of the user.

8. The system for gesture-based human-machine interaction according to claim 7, wherein the capturing module captures the video stream through a video camera, and captures continuous images from the video stream, or captures discrete images from the video stream in a predetermined interval.

9. The system for gesture-based human-machine interaction according to claim 8, wherein the positioning module comprises:

a background segmentation unit configured to set up a background model for generating a mask picture, and to remove the background of the images by the mask picture;

a histogram unit configured to obtain predetermined templates of a first color block, a second color block and a third color block of the three or more predetermined color blocks, and to generate a histogram corresponding to each of the first, second and third color blocks according to the images, wherein the first color block corresponds to one finger of the one hand of the user, and the second and third color blocks respectively correspond to two fingers of the other hand of the user;

a probability distribution diagram unit configured to calculate a probability distribution diagram of each of the first, second and third color blocks according to the corresponding histogram; and

a trace unit configured to trace a centroid of each of the first, second and third color blocks in the probability distribution diagram, to determine the coordinates of the first, second and third color blocks.

10. The system for gesture-based human-machine interaction according to claim 9, wherein the transforming module is configured to determine the straight line distance between the coordinates of the second color block and the third color block, and to determine if the straight line distance is less than a predetermined threshold value;

to simulate a press action of a left key of the mouse if the straight line distance is less than the predetermined threshold value; and

to simulate a release action of the left key of the mouse if the straight line distance is not less than the predetermined threshold value.

11. The system according to claim 10, wherein the system further comprises,

three sleeves with three different colors wore on three different fingers of the user, wherein the colors of the three sleeves are stored correspondingly as the predetermined templates for the first, second and third color blocks.

12. The system according to claim 9, wherein the positioning module further comprises an optimization unit, wherein the optimization unit comprises a noise removing subunit and a smoothing subunit, wherein the noise removing subunit is used to remove noise by applying noise erosion operation on the probability distribution diagrams, and the smoothing subunit is used to apply a Gaussian smoothing process on the probability distribution diagrams, and executing threshold segmentation on the Gaussian smoothed diagrams.

13. At least one non-transitory computer-readable medium having computer-executive instructions stored thereon that, when executed by a processor, cause a computer to implement a gesture-based human-machine interaction method, the method comprising:

capturing images from a video stream of a user, wherein each of the images comprises a foreground and a background, and the foreground comprises three or more predetermined color blocks, wherein the three or more predetermined color blocks comprises at least one color block corresponding to one hand of the user, and at least two color blocks corresponding to the other hand of the user;

positioning coordinates of the three or more predetermined color blocks in the foreground of the images; and

simulating movements of a mouse according to the coordinates of the at least one color block corresponding to the one hand of the user, and simulating click actions of the mouse according to the coordinates of the at least two color blocks corresponding to the other hand of the user.

14. The non-transitory computer-readable medium according to claim 13, wherein the step of capturing images comprises:

capturing the video stream using a video camera, and capturing continuous images from the video stream, or capturing discrete images from the video stream in a predetermined interval.

15. The non-transitory computer-readable medium according to claim 14, wherein the step of positioning coordinates comprises:

setting up a background model for generating a mask picture, and removing the background of the images by the mask picture;

obtaining predetermined templates of a first color block, a second color block and a third color block of the three or more predetermined color blocks, wherein the first color block corresponds to one finger of the one hand of the user, and the second and third color blocks respectively correspond to two fingers of the other hand of the user;

generating a histogram corresponding to each of the first, second and third color blocks according to the images;

calculating a probability distribution diagram of each of the first, second and third color blocks according to the corresponding histogram;

tracing a centroid of each of the first, second and third color blocks in the probability distribution diagram to determine the coordinates of the first, second and third color blocks.

16. The non-transitory computer-readable medium according to claim 15, wherein the step of simulating click actions of the mouse according to the coordinates of the at least two color blocks corresponding to the other hand of the user comprises:

determining the straight line distance between the coordinates of the second color block and the third color block,

determining if the straight line distance is less than a predetermined threshold value;

simulating a press action of a left key of the mouse if the straight line distance is less than the predetermined threshold value; and

simulating a release action of the left key of the mouse if the straight line distance is not less than the predetermined threshold value.

17. The non-transitory computer-readable medium according to claim 16, wherein the step of obtaining predetermined templates comprises:

providing three sleeves three different colors on three different fingers of the user, and

storing the colors of the three sleeves correspondingly as the predetermined templates for the first, second and third color blocks.

18. The non-transitory computer-readable medium according to claim 15, wherein the method further comprises the following steps after the step of calculating the probability distribution diagram:

removing noise from the probability distribution diagrams by applying a noise erosion operation;

applying a Gaussian smoothing on the probability distribution diagrams, and executing threshold segmentation on the Gaussian smoothed probability distribution diagrams.