Gesture manipulation device and method, program, and recording medium

• US 9,939,909 B2
• Filed: 05/15/2014
• Issued: 04/10/2018
• Est. Priority Date: 09/12/2013
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Corporation
• Original Assignee: Mitsubishi Electric Corporation
• Inventors: Yamagishi, Nobuhiko, Nakamura, Yudai, Fukuta, Tomonori
• Primary Examiner(s): Ritchie, Darlene M
• Application Number:
  US14/908,513
• Publication Number:
  US 20160209927A1
• Time in Patent Office: 1,426 Days
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A hand region is detected in a captured image, and for each part of the background area, a light source presence degree indicating the probability that a light source is present is determined according to the luminance or color of that part; on the basis of the light source presence degree, a region in which the captured image is affected by a light source is estimated, and if the captured image includes a region estimated to be affected by a light source, whether or not a dropout has occurred in the hand region in the captured image is decided; on the basis of the result of this decision, an action is determined. Gesture determinations can be made correctly even when the hand region in the captured image is affected by a light source at the time of gesture manipulation input.

Claims

1. A manipulation input device comprising:

a processor; and

a memory storing a program which, when executed, causes a computer to perform a process including,

detecting a hand region in a captured image obtained by imaging by an image capture unit;

setting a light source presence degree as an index indicating a probability that a light source is present, for each part of a background region comprising a region other than the detected hand region, on a basis of luminance or color of the part;

estimating, on a basis of the light source presence degree, a light source region which is a region affected by a light source in the captured image, and determining whether or not the estimated light source region and the detected hand region are within a predetermined distance of each other;

making a decision, on a basis of a shape of the detected hand region, as to whether or not a dropout has occurred in the hand region in the captured image, when the estimated light source region and the hand region are determined to be within the predetermined distance of each other; and

making a determination of an action of a hand on a basis of a result of the decision made and information representing the detected hand region,

wherein, when the decision is made that the dropout occurred in the detected hand region in the captured image, the determination of the action of the hand is made by

correcting the shape of the detected hand region, and

generating action information indicating the action of the hand on a basis of the corrected shape of the detected hand region.

2. The manipulation input device of claim 1, wherein

the image obtained by the image capture unit is a moving image comprising a series of frames, each representing a still image;

on a basis of information representing the estimated light source region in the current frame, and information about a feature point in the hand region in at least one temporally preceding frame, an estimate is made of a position of the feature point in the hand region in the current frame, in the estimated light source region in the current frame; and

the shape of the detected hand region is corrected on a basis of a result of the estimate.

3. The manipulation input device of claim 1, wherein the process further comprises giving a manipulation instruction corresponding to the action of the hand determined on a basis of a result of the determination made of the action of the hand.

4. The manipulation input device of claim 1, wherein the decision on whether or not the dropout occurred is made by:

detecting a position of a center of a palm;

determining a distance from the detected center of the palm to a perimeter of the hand region in each direction; and

deciding whether or not the dropout has occurred in the hand region in the captured image on a basis of a relation between the distance in each direction and the direction.

5. The manipulation input device of claim 4, wherein the decision on whether or not the dropout occurred is made by:

estimating a probability that the relation between the distance in each direction and the direction corresponds to a shape of the hand made when a gesture for manipulation input is performed; and

deciding whether or not the dropout has occurred in the hand region in the captured image on a basis of the estimated probability.

6. The manipulation input device of claim 4, wherein the decision on whether or not a dropout has occurred in the hand region in the captured image is made on a basis of a size of a direction range having only one local minimum point on a curve representing the relation between the distance in each direction and the direction.

7. The manipulation input device of claim 6, wherein said direction range is such that, when a local maximum point is present in an interior of the direction range on the curve representing the relation between the distance in each direction and the direction, a difference between the local maximum point and the local minimum point is smaller than a predetermined threshold value.

8. The manipulation input device of claim 6, wherein said direction range is such that, in the interior of the direction range, a magnitude of a change in the distance with respect to a change in the direction is equal to or less than a predetermined value.

9. The manipulation input device of claim 1, the process further comprising giving a manipulation instruction corresponding to the action of the hand determined on a basis of a result of the determination made on the action of the hand.

10. The manipulation input device of claim 9, wherein, when the process decides that the dropout has occurred in the hand region in the captured image, the process informs an operator that a dropout has occurred in the hand region in the captured image.

11. A manipulation input method comprising:

detecting a hand region in a captured image obtained by imaging by an image capture step;

setting a light source presence degree as an index indicating a probability that a light source is present, for each part of a background region comprising a region other than the detected hand region, on a basis of luminance or color of the part;

estimating, on a basis of the light source presence degree, a light source region which is a region affected by a light source in the captured image, and determining whether or not the estimated light source region and the detected hand region are within a predetermined distance of each other;

making a decision, on a basis of a shape of the detected hand region, as to whether or not a dropout has occurred in the hand region in the captured image, when the estimated light source region and the hand region are determined to be within the predetermined distance of each other; and

making a determination of an action of a hand on a basis of a result of the decision made and information representing the detected hand region,

wherein, when the decision is made that the dropout occurred in the detected hand region in the captured image, the determination of the action of the hand is made by

correcting the shape of the detected hand region, and

generating action information indicating the action of the hand on a basis of the corrected shape of the detected hand region.

12. A non-transitory computer readable recording medium in which a program for causing a computer to execute the steps in the manipulation input method as set forth in claim 11 is recorded.