IMAGE PROCESSING APPARATUS, METHOD, AND PROGRAM

• US 20120114255A1
• Filed: 10/26/2011
• Published: 05/10/2012
• Est. Priority Date: 11/04/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: KIMURA, Jun
• Granted Patent: US 9,792,494 B2
• Time in Patent Office: Days
• US Class Current: 382/203
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

An image processing apparatus includes a facial image detection unit which detects a facial image from an input image; a posture estimation unit which estimates a posture of a person in the input image from a position of the facial image; a hand position detection unit which detects positions of hands of the person based on the posture; a hand image extraction unit which extracts a hand image of the person from the input image based on information regarding the positions of the hands of the person; a hand shape specifying unit which specifies shapes of the hands of the person based on the hand image; a hand shape time-series storage unit which stores the shapes of the hands in a time-series; and a hand gesture recognition unit which recognizes a hand gesture based on information regarding the shapes of the hands.

Claims

1. An image processing apparatus comprising:

a facial image detection unit which detects a facial image from an input image;

a posture estimation unit which estimates a posture of a person in the input image from a position of the facial image detected by the facial image detection unit;

a hand position detection unit which detects positions of hands of the person based on the posture estimated by the posture estimation unit;

a hand image extraction unit which extracts a hand image of the person from the input image based on information regarding the positions of the hands of the person detected by the hand position detection unit;

a hand shape specifying unit which specifies shapes of the hands of the person based on the hand image extracted by the hand image extraction unit;

a hand shape time-series storage unit which stores the shapes of the hands specified by the hand shape specifying unit in a time-series by corresponding to the information regarding the positions of the hands; and

a hand gesture recognition unit which recognizes a hand gesture, which is a gesture using the hands of the person, based on information regarding the shapes of the hands stored in the time-series in the hand shape time-series storage unit by corresponding to the information regarding the positions of the hands.

2. The image processing apparatus according to claim 1, further comprising:

a hand gesture storage unit which stores the information regarding the shapes of the hands in a time-series by corresponding to the information regarding the positions of the hands for each type of hand gesture,

wherein the hand gesture recognition unit compares the information regarding the shapes of the hands stored in the time-series in the hand shape time-series storage unit by corresponding to the information regarding the positions of the hands, to the information regarding the shapes of the hands stored in the time-series in the hand gesture storage unit by corresponding to the information regarding the positions of the hands for each hand gesture, and then recognizes the hand gesture based on the comparison result.

3. The image processing apparatus according to claim 2, further comprising:

a similarity calculation unit which calculates similarity between the information regarding the shapes of the hands stored in the time-series in the hand shape time-series storage unit by corresponding to the information regarding the positions of the hands and the information regarding the shapes of the hands stored in the time-series in the hand gesture storage unit by corresponding to the information regarding the positions of the hands for each hand gesture,

wherein the hand gesture recognition unit sets the hand gesture with the highest similarity as a recognition result.

4. The image processing apparatus according to claim 3, wherein the hand gesture recognition unit considers that the hand gesture as the recognition result is not present, when the highest similarity is less than a predetermined threshold value.

5. The image processing apparatus according to claim 3, further comprising:

a registration unit which registers the information regarding the shapes of the hands stored in the time-series in the hand shape time-series storage unit by corresponding to the information regarding the positions of the hands, in the hand gesture storage unit by corresponding to a predetermined hand gesture, when the highest similarity is less than a predetermined threshold value.

6. An image processing method of an image processing apparatus including a facial image detection unit which detects a facial image from an input image, a posture estimation unit which estimates a posture of a person in the input image from a position of the facial image detected by the facial image detection unit, a hand position detection unit which detects positions of hands of the person based on the posture estimated by the posture estimation unit, a hand image extraction unit which extracts a hand image of the person from the input image based on information regarding the positions of the hands of the person detected by the hand position detection unit, a hand shape specifying unit which specifies shapes of the hands of the person based on the hand image extracted by the hand image extraction unit, a hand shape time-series storage unit which stores the shapes of the hands specified by the hand shape specifying unit in a time-series by corresponding to the information regarding the positions of the hands, and a hand gesture recognition unit which recognizes a hand gesture, which is a gesture using the hands of the person, based on information regarding the shapes of the hands stored in the time-series in the hand shape time-series storage unit by corresponding to the information regarding the positions of the hands, the image processing method comprising:

detecting, by the facial image detection unit, the facial image from the input image;

estimating, by the posture estimation unit, the posture of the person in the input image from the position of the facial image detected in the detecting of the facial image;

detecting, by the hand position detection unit, the positions of the hands of the person based on the posture estimated in the estimating of the posture of the person;

extracting, by the hand image extraction unit, the hand image of the person from the input image based on the information regarding the positions of the hands of the person detected in the detecting of the hand positions of the hands;

specifying, by the hand shape specifying unit, the shapes of the hands of the person based on the hand image extracted in the extracting of the hand image;

storing, by the hand shape time-series storage unit, the shapes of the hands specified in the specifying of the shapes of the hands in the time-series by corresponding to the information regarding the positions of the hands; and

recognizing, by the hand gesture recognition unit, the hand gesture, which is a gesture using the hands of the person, based on the information regarding the shapes of the hands stored in the time-series in the storing of the shapes of the hands by corresponding to the information regarding the positions of the hands.

7. A program causing a computer, which controls an image processing apparatus including a facial image detection unit which detects a facial image from an input image, a posture estimation unit which estimates a posture of a person in the input image from a position of the facial image detected by the facial image detection unit, a hand position detection unit which detects positions of hands of the person based on the posture estimated by the posture estimation unit, a hand image extraction unit which extracts a hand image of the person from the input image based on information regarding the positions of the hands of the person detected by the hand position detection unit, a hand shape specifying unit which specifies shapes of the hands of the person based on the hand image extracted by the hand image extraction unit, a hand shape time-series storage unit which stores the shapes of the hands specified by the hand shape specifying unit in a time-series by corresponding to the information regarding the positions of the hands, and a hand gesture recognition unit which recognizes a hand gesture, which is a gesture using the hands of the person, based on information regarding the shapes of the hands stored in the time-series in the hand shape time-series storage unit by corresponding to the information regarding the positions of the hands, to execute

detecting, by the facial image detection unit, the facial image from the input image,

estimating, by the posture estimation unit, the posture of the person in the input image from the position of the facial image detected in the detecting of the facial image,

detecting, by the hand position detection unit, the positions of the hands of the person based on the posture estimated in the estimating of the posture of the person,

extracting, by the hand image extraction unit, the hand image of the person from the input image based on the information regarding the positions of the hands of the person detected in the detecting of the hand positions of the hands,

specifying, by the hand shape specifying unit, the shapes of the hands of the person based on the hand image extracted in the extracting of the hand image,

storing, by the hand shape time-series storage unit, the shapes of the hands specified in the specifying of the shapes of the hands in the time-series by corresponding to the information regarding the positions of the hands, and

recognizing, by the hand gesture recognition unit, the hand gesture, which is a gesture using the hands of the person, based on the information regarding the shapes of the hands stored in the time-series in the storing of the shapes of the hands by corresponding to the information regarding the positions of the hands.