Apparatus for controlling virtual mouse based on hand motion and method thereof

• US 9,430,039 B2
• Filed: 07/07/2014
• Issued: 08/30/2016
• Est. Priority Date: 07/15/2013
• Status: Active Grant

• Current Assignee: Korea Electronics Technology Institute (Government of South Korea)
• Original Assignee: Korea Electronics Technology Institute (Government of South Korea)
• Inventors: Ahn, Yang Keun, Jung, Kwang Mo
• Primary Examiner(s): SHAH, PRIYANK J
• Application Number:
  US14/324,447
• Publication Number:
  US 20150015490A1
• Time in Patent Office: 785 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Provided are an apparatus and method for controlling a virtual mouse based on a hand motion. The apparatus includes an image processing unit configured to measure a maximum horizontal width of a hand region of a user and a width of an index finger in any one of a first depth image captured in a state in which the index finger is spread out in a vertical direction with his or her fist and a second depth image captured in a state in which the index finger is folded with the fist, a hand motion recognizing unit configured to compare the maximum horizontal width and the width of the index finger to recognize whether the index finger is folded, and a function matching unit configured to match a change in a folded state of the index finger to a pre-set first function and output a control signal for the function.

Claims

1. An apparatus for controlling a virtual mouse based on a hand motion, the apparatus comprising:

an image processing unit configured to measure a maximum horizontal width of a hand region of a user and a width of a finger in any one of a first depth image captured in a state in which the finger is spread out in a vertical direction with his or her fist and a second depth image captured in a state in which the finger is folded with the fist;

a hand motion recognizing unit configured to compare the maximum horizontal width and the width of the finger to recognize whether the finger is folded; and

a function matching unit configured to match a change in a folded state of the finger to a pre-set first function and output a control signal for implementing the pre-set first function,

wherein the image processing unit detects the hand region of the user using foreground/background separation from the first depth image and the second depth image, generates a horizontal histogram from a hand region image of the user, and measures the maximum horizontal width at a point having a maximum pixel number.

2. The apparatus of claim 1, wherein the image processing unit determines a region of interest by a predefined length specified from an upper end of the horizontal histogram, and determines an average width of the hand region within the region of interest, as the width of the finger.

3. The apparatus of claim 1, wherein the hand motion recognizing unit compares the width of the finger with a width corresponding to a predefined percentage of the maximum horizontal width, and when the width of the finger is less than the width corresponding to the predefined percentage, the hand motion recognizing unit recognizes the finger in a spread state, and when the width of the finger is equal to or greater than the width corresponding to the predefined percentage, the hand motion recognizing unit recognizes the finger in the folded state.

4. The apparatus of claim 1, wherein the image processing unit generates a distance-transformed image in units of pixels with respect to an image of the hand region of the user, and detects a pixel having the greatest value in the distant-transformed image, as a marker.

5. The apparatus of claim 4, wherein the hand motion recognizing unit recognizes a change in a position of the hand using the marker, and the function matching unit matches the change in the position of the hand to a pre-set second function and outputs a control signal for implementing the pre-set second function.

6. The apparatus of claim 1, wherein the image processing unit comprises:

a foreground/background separating unit configured to separate a foreground and a background on the basis of the depth information from the first depth image and the second depth image;

a horizontal hand width measuring unit configured to measure the maximum horizontal width of the hand region image of the user from which the background has been separated;

a horizontal finger width measuring unit configured to measure the finger width in the hand region image of the user; and

a marker detecting unit configured to detect a marker to be used to recognize a change in a position of the hand from the hand region image of the user.

7. A method for controlling a virtual mouse based on a hand motion, the method comprising:

measuring a maximum horizontal width of a hand region of a user and a width of a finger in any one of a first depth image captured in a state in which the finger is spread out in a vertical direction with his or her fist and a second depth image captured in a state in which the finger is folded with the fist;

comparing the maximum horizontal width and the width of the finger to recognize whether the finger is folded; and

matching a change in a folded state of the finger to a pre-set first function and outputting a control signal for implementing the pre-set first function,

wherein the measuring of the maximum horizontal width and the width of the finger comprises;

detecting the hand region of the user using foreground/background separation from the first depth image and the second depth image, generating a horizontal histogram from a hand region image of the user, and measuring the maximum horizontal width at a point having a maximum pixel number.

8. The method of claim 7, wherein the measuring of the maximum horizontal width and the width of the finger comprises:

determining a region of interest by a predefined length specified from an upper end of the horizontal histogram, and determining an average width of the hand region within the region of interest, as the width of the finger.

9. The method of claim 8, further comprising:

generating a distance-transformed image in units of pixels with respect to the image of the hand region of the user, and detecting a pixel having the greatest value in the distant-transformed image, as a marker.

10. The method of claim 7, wherein the recognizing of whether the finger is folded comprises:

comparing the width of the finger with a width corresponding to a predefined percentage of the maximum horizontal width; and

when the width of the finger is less than the width corresponding to the predefined percentage, recognizing the finger in a spread state, and when the width of the finger is equal to or greater than the width corresponding to the predefined percentage, recognizing the finger in the folded state.

11. The method of claim 9 further comprising:

recognizing a change in a position of the hand using the marker; and

matching the change in the position of the hand to a pre-set second function and outputs a control signal for implementing the pre-set second function.