GESTURE MANIPULATION DEVICE AND METHOD, PROGRAM, AND RECORDING MEDIUM

• US 20160209927A1
• Filed: 05/15/2014
• Published: 07/21/2016
• Est. Priority Date: 09/12/2013
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Corporation
• Original Assignee: Mitsubishi Electric Corporation
• Inventors: YAMAGISHI, Nobuhiko, NAKAMURA, Yudai, FUKUTA, Tomonori
• Granted Patent: US 9,939,909 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A hand region is detected in a captured image, and for each part of the background area, a light source presence degree indicating the probability that a light source is present is determined according to the luminance or color of that part; on the basis of the light source presence degree, a region in which the captured image is affected by a light source is estimated, and if the captured image includes a region estimated to be affected by a light source, whether or not a dropout has occurred in the hand region in the captured image is decided; on the basis of the result of this decision, an action is determined. Gesture determinations can be made correctly even when the hand region in the captured image is affected by a light source at the time of gesture manipulation input.

Claims

1. A manipulation input device comprising:

a hand region detection unit for detecting a hand region in a captured image obtained by imaging by an image capture unit;

a light source presence degree information generating unit for setting a light source presence degree as an index indicating a probability that a light source is present, for each part of a background region comprising a region other than the hand region detected by the hand region detection unit, on a basis of luminance or color of the part;

a light source estimation unit for estimating, on a basis of the light source presence degree, a light source region which is a region affected by a light source in the captured image, and determining whether or not the estimated light source region and the hand region detected by the hand region detection unit are within a predetermined distance of each other;

a shape analysis/evaluation unit for making a decision, on a basis of a shape of the hand region detected by the hand region detection unit, as to whether or not a dropout has occurred in the hand region in the captured image, when the estimated light source region and the hand region are determined to be within the predetermined distance of each other; and

an action determination unit for making a determination of an action of a hand on a basis of a result of the decision made by the shape analysis/evaluation unit and information representing the hand region detected by the hand region detection unit.

2. The manipulation input device of claim 1, wherein the action determination unit comprises:

a shape correction unit for correcting the shape of the hand region detected by the hand region detection unit when the shape analysis/evaluation unit decides that a dropout has occurred in the hand region in the captured image; and

an action information generating unit for generating action information indicating the action of the hand on a basis of the shape of the hand region having been corrected by the shape correction unit.

3. The manipulation input device of claim 1, wherein the shape analysis/evaluation unit:

detects a position of a center of a palm;

determines a distance from the detected center of the palm to a perimeter of the hand region in each direction; and

decides whether or not a dropout has occurred in the hand region in the captured image on a basis of a relation between the distance in each direction and the direction.

4. The manipulation input device of claim 3, wherein the shape analysis/evaluation unit:

estimates a probability that the relation between the distance in each direction and the direction corresponds to a shape of the hand made when a gesture for manipulation input is performed; and

decides whether or not a dropout has occurred in the hand region in the captured image on a basis of the estimated probability.

5. The manipulation input device of claim 3, wherein the shape analysis/evaluation unit decides whether or not a dropout has occurred in the hand region in the captured image on a basis of a size of a direction range having only one local minimum point on a curve representing the relation between the distance in each direction and the direction.

6. The manipulation input device of claim 5, wherein said direction range is such that, when a local maximum point is present in an interior of the direction range on the curve representing the relation between the distance in each direction and the direction, a difference between the local maximum point and the local minimum point is smaller than a predetermined threshold value.

7. The manipulation input device of claim 5, wherein said direction range is such that, in the interior of the direction range, a magnitude of a change in the distance with respect to a change in the direction is equal to or less than a predetermined value.

8. The manipulation input device of claim 2, wherein

the image obtained by the image capture unit is a moving image comprising a series of frames, each representing a still image;

on a basis of information representing the estimated light source region in the current frame, and information about a feature point in the hand region in at least one temporally preceding frame, the shape correction unit makes an estimate of a position of the feature point in the hand region in the current frame, in the estimated light source region in the current frame; and

the shape correction unit performs the correction on a basis of a result of the estimate.

9. The manipulation input device of claim 8, wherein

the shape analysis/evaluation unit detects a center position of a palm;

when, among pixels on a line segment joining the estimated position of the feature point and the detected center position of the palm, all the pixels except pixels which are within the predetermined distance from the hand region are positioned in the light source region or the hand region, the shape analysis/evaluation unit recognizes the estimated position of the feature point as a corrected position of the feature point; and

when among the pixels on the line segment joining the estimated position of the feature point and the detected center position of the palm, any one or more of all the pixels except the pixels which are within the predetermined distance from the hand region are not included in either of the light source region and the hand region, the shape analysis/evaluation unit recognizes, as the corrected position of the feature point, a position of a pixel that is adjacent to a pixel closest to the center position of the palm, among the pixels on the line segment not included in either of the light source region and the hand region, and that is positioned on the side of the center position of the palm.

10. The manipulation input device of claim 1, further comprising a manipulation content determination unit for giving a manipulation instruction corresponding to the action of the hand determined on a basis of a result of the determination made by the action determination unit.

11. The manipulation input device of claim 10, wherein, when the shape analysis/evaluation unit decides that a dropout has occurred in the hand region in the captured image, the action determination unit furnishes the manipulation content determination unit with information to that effect, on a basis of which the manipulation content determination unit gives a manipulation instruction for informing an operator that a dropout has occurred in the hand region in the captured image.

12. The manipulation input device of claim 1, wherein, when the shape analysis/evaluation unit decides that a dropout has occurred in the hand region in the captured image, the action determination unit does not make the determination of the action of the hand based on that captured image.

16. The manipulation input device of claim 2, wherein the shape analysis/evaluation unit:

detects a position of a center of a palm;

determines a distance from the detected center of the palm to a perimeter of the hand region in each direction; and

decides whether or not a dropout has occurred in the hand region in the captured image on a basis of a relation between the distance in each direction and the direction.

17. The manipulation input device of claim 2, further comprising a manipulation content determination unit for giving a manipulation instruction corresponding to the action of the hand determined on a basis of a result of the determination made by the action determination unit.

13. A manipulation input method comprising:

a hand region detection step for detecting a hand region in a captured image obtained by imaging by an image capture step;

a light source presence degree information generating step for setting a light source presence degree as an index indicating a probability that a light source is present, for each part of a background region comprising a region other than the hand region detected by the hand region detection step, on a basis of luminance or color of the part;

a light source estimation step for estimating, on a basis of the light source presence degree, a light source region which is a region affected by a light source in the captured image, and determining whether or not the estimated light source region and the hand region detected by the hand region detection step are within a predetermined distance of each other;

a shape analysis/evaluation step for making a decision, on a basis of a shape of the hand region detected by the hand region detection step, as to whether or not a dropout has occurred in the hand region in the captured image, when the estimated light source region and the hand region are determined to be within the predetermined distance of each other; and

an action determination step for making a determination of an action of a hand on a basis of a result of the decision made by the shape analysis/evaluation step and information representing the hand region detected by the hand region detection step.

15. A computer readable recording medium in which a program for causing a computer to execute the steps in the manipulation input method as set forth in claim 13 is recorded.

14. (canceled)