Image processing device, image device, image processing method

• US 8,515,136 B2
• Filed: 01/12/2006
• Issued: 08/20/2013
• Est. Priority Date: 07/18/2003
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Mori, Katsuhiko, Kaneda, Yuji, Matsugu, Masakazu, Mitarai, Yusuke, Suzuki, Takashi
• Primary Examiner(s): TUCKER, WESLEY J
• Application Number:
  US11/330,138
• Publication Number:
  US 20060115157A1
• Time in Patent Office: 2,777 Days
• Field of Search: 382/115, 382/118, 382/116, 382/117, 382/181, 382/190, 382/209, 382/224
• US Class Current: 382/118
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

An image including a face is input (S201), a plurality of local features are detected from the input image, a region of a face in the image is specified using the plurality of detected local features (S202), and an expression of the face is determined on the basis of differences between the detection results of the local features in the region of the face and detection results which are calculated in advance as references for respective local features in the region of the face (S204).

Claims

1. An image sensing apparatus comprising:

a sensing unit constructed to successively sense images;

a region determination unit constructed to determine a face region in each of the successively sensed images;

an extraction unit constructed to extract predetermined features representing parts of a face from the determined face region in each of the successively sensed images;

a score derivation unit constructed to derive a score corresponding to an expression of the face in each of the successively sensed images on the basis of the predetermined features extracted from the face region;

a recording unit constructed to selectively record the successively sensed images in a predetermined storage device; and

a control unit constructed to control whether or not to perform recording in the predetermined storage device for each of the successively sensed images, based on the score derived for each of the successively sensed images.

9. An image sensing apparatus according to claim 1, further comprising:

a first feature detection unit for detecting a plurality of local features from an entirety of the image sensed by said sensing unit; and

a second feature detection unit for detecting predetermined features representing parts of the face from the entirety of the sensed image on the basis of combinations of the detected local features,

wherein said region determination unit determines the face region in the sensed image on the basis of the detected predetermined features.

10. An image sensing apparatus according to claim 9, wherein said extraction unit extracts predetermined features included within the determined predetermined face region from the predetermined features in the entirety of the sensed image detected by said second feature detection unit.

11. An image sensing apparatus according to claim 1, wherein said score derivation unit derives the score corresponding to the expression of the face on the basis of the differences between relative positions of the predetermined features extracted from the face region and predetermined features detected in a face image which is set in advance as a reference.

13. The apparatus according to claim 9, wherein said first feature detection unit, said second feature detection unit and said region determination unit comprise a hierarchical neural network, and wherein intermediate layer outputs of the hierarchical neural network are used as detection results for the predetermined features.

14. The apparatus according to claim 11, wherein said score derivation unit derives which distribution, of a number of distributions corresponding to respective expressions derived in advance, has a highest similarity to a distribution of differences between relative positions of the predetermined features extracted from the face region and relative positions derived in advance as references for the predetermined features in the face region, and determines an expression indicated by the distribution with the highest similarity.

15. The apparatus according to claim 11, wherein

said input unit successively inputs images by successively executing a process of inputting a next image when said region determination unit completes a process for determining a face region, and

wherein said score derivation unit executes a process for deriving the score corresponding to the expression of the face on the basis of differences between relative positions of the predetermined features extracted from the face region using an image previously sensed by said sensing unit at a timing when said sensing unit inputs an image, and relative positions derived in advance as references for the predetermined features in the face region.

2. An image processing apparatus comprising:

a sensing unit constructed to successively sense frame images;

a region determination unit constructed to determine a face region in each of the successively sensed frame images;

an extraction unit constructed to extract predetermined features representing parts of a face from the determined face region in each of the successively sensed frame images;

a score derivation unit constructed to derive a score corresponding to an expression of the face on the basis of differences between relative positions of predetermined features detected from a face image which is set in advance as a reference and the extracted predetermined features included in a region of an image of a second frame subsequent to a first frame, wherein the region of the image of the second frame positionally corresponds to the face region determined by said region determination unit in an image of the first frame sensed by said sensing unit;

a recording unit constructed to selectively record the successively sensed frame images in a predetermined storage device; and

a control unit constructed to control whether or not to perform recording in the predetermined storage device for each of the successively sensed frame images, based on the score derived for each of the successively sensed frame images.

3. An image processing apparatus comprising:

a sensing unit constructed to successively sense images;

a region determination unit constructed to determine a face region in each of the successively sensed images;

an extraction unit constructed to extract predetermined features representing parts of a face from the determined face region in each of the successively sensed images;

a first determination unit constructed to identify a person who has the face in the image sensed by said sensing unit using the predetermined features extracted from the face region determined by said region determination unit;

score derivation unit constructed to derive a score corresponding to an expression of the face using the predetermined features extracted from the face region determined by said region determination unit;

a recording unit constructed to selectively record the successively sensed images in a predetermined storage device; and

a control unit constructed to control whether or not to perform recording in the predetermined storage device for each of the successively sensed images, based on the score derived for each of the successively sensed images.

12. An image processing apparatus according to claim 3, further comprising:

a first feature detection unit for detecting a plurality of local features from an entirety of the image sensed by said sensing unit; and

a second feature detection unit for detecting predetermined features representing parts of the face from the entirety of the sensed image on the basis of combination of the detected local features,

wherein said region determination unit determines the face region in the sensed image on the basis of the detected predetermined features.

16. An image processing apparatus according to claim 3, wherein said score derivation unit changes parameters used to perform expression determination for a face of interest, in according with a result from first determination unit which identifies the person of the face of interest.

4. An image processing method comprising:

performing by a processor the following steps;

an input step of successively sensing images;

a region determination step of determining a face region in each of the successively sensed images;

an extraction step of extracting predetermined features representing parts of a face from the determined face region in each of the successively sensed images;

a score derivation step of deriving a score corresponding to an expression of the face in each of the successively sensed images on the basis of the predetermined features extracted from the face region;

a recording step of selectively recording the successively sensed images in a predetermined storage device; and

a control step of controlling whether or not to perform recording in the predetermined storage device for each of the successively sensed images, based on the score derived for each of the successively sensed images.

7. A computer-executable program retrievably stored on a non-transitory computer-readable storage medium for making a computer execute an image processing method of claim 4.

8. A non-transitory computer-readable storage medium retrievably storing a program of claim 7.

5. An image processing method comprising:

performing by a processor the following steps;

an input step of successively sensing frame images;

a region determination step of determining a face region in each of the successively sensed frame images;

an extraction step of extracting predetermined features representing parts of a face from the determined face region in each of the successively sensed frame images;

a score derivation step of deriving a score corresponding to an expression of the face on the basis of differences between relative positions of the predetermined features detected from a face image which is set in advance as a reference and the extracted predetermined features included in a region of an image of a second frame subsequent to a first frame, wherein the region of the image of the second frame positionally corresponds to the face region determined in said region determination step in an image of the first frame sensed in the sensing step;

a recording step of selectively recording the successively sensed frame images in a predetermined storage device; and

a control step of controlling whether or not to perform recording in the predetermined storage device for each of the successively sensed frame images, based on the score derived for each of the successively sensed frame images.

17. A non-transitory computer-readable storage medium retrievably storing a program for making a computer execute an image processing method of claim 5.

6. An image processing method comprising:

performing by a processor the following steps;

a sensing step of successively sensing images;

a region determination step of determining a face region in each of the successively sensed images;

an extraction step of extracting predetermined features representing parts of a face from the determined face region in each of the successively sensed images;

a first determination step of identifying a person who has the face in the image sensed in the sensing step using the predetermined features extracted from the face region determined in the region determination step;

a score derivation step of deriving a score corresponding to an expression of the face using the predetermined features extracted from the face region determined in the region determination step;

a recording step of selectively recording the successively sensed images in a predetermined storage device; and

a controlling step of controlling whether or not to perform recording in the predetermined storage device for each of the successively sensed images, based on the score derived for each of the successively sensed images.

18. A non-transitory computer-readable storage medium retrievably storing a program for making a computer execute an image processing method of claim 6.