Information processing apparatus and control method therefor

• US 8,542,928 B2
• Filed: 09/19/2006
• Issued: 09/24/2013
• Est. Priority Date: 09/26/2005
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kaneda, Yuji, Matsugu, Masakazu, Mori, Katsuhiko
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): Li, Ruiping
• Application Number:
  US11/532,979
• Publication Number:
  US 20070122036A1
• Time in Patent Office: 2,562 Days
• Field of Search: 382/190, 382/118, 382/108, 382/100, 382/101, 382/103, 382/195, 382/144, 382/155
• US Class Current: 382/190
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

An information processing apparatus includes an image input unit which inputs image data containing a face, a face position detection unit which detects, from the image data, the position of a specific part of the face, and a facial expression recognition unit which detects a feature point of the face from the image data on the basis of the detected position of the specific part and determines facial expression of the face on the basis of the detected feature point. The feature point is detected at a detection accuracy higher than detection of the position of the specific part. Detection of the position of the specific part is robust to a variation in the detection target.

Claims

1. An information processing apparatus comprising:

an input device adapted to input image data containing a face;

a memory adapted to store the image data;

a first detection device adapted to detect, from the image data stored in said memory, positions of a plurality of specific parts of the face, wherein said first detection device has been learned by supervisory data with various kinds of variations;

an area setting device adapted to set respective feature point pixel extraction areas for a plurality of feature point pixels of the specific parts in the image data on the basis of the detected positions of the specific parts;

a second detection device adapted to detect respective positions of the plurality of feature point pixels of the specific parts from the respective feature point pixel extraction areas, wherein said second detection device has been learned by supervisory data with specific variation that is one of the various kinds of variations; and

a determination device adapted to determine facial expression of the face on the basis of the detected feature point pixels,

wherein said second detection device has higher detection accuracy than detection accuracy of said first detection device, and said first detection device is robust to a variation in a detection target.

2. The apparatus according to claim 1, wherein said first detection device detects the position of the specific part of the face on the basis of image data with a lower resolution than image data to be processed by said second detection device.

3. The apparatus according to claim 1, further comprising a storage device adapted to store image data of the face with predetermined facial expression,

wherein said determination device determines first facial expression on the basis one of a difference and a ratio between a first feature amount calculated by using the feature point pixel in the input image data and a second feature amount calculated by using the feature point pixel in the image data stored in said storage device.

4. The apparatus according to claim 3, wherein said determination device executes the determination after multiplying the feature amount by a preset weighting value.

5. The apparatus according to claim 1, wherein

the input image data includes a moving image containing a plurality of frames,

said first detection device detects the position of the specific part of the face in each of the frames,

said second detection device detects the feature point pixel of the face in each of the frames, and

said determination device determines a change in the facial expression of the face on the basis of a variation in the feature point pixel between frames adjacent to each other.

6. The apparatus according to claim 1, further comprising a correction device adapted to correct luminance of the input image data to enhance tone of a low-luminance region.

7. The apparatus according to claim 1, wherein the specific part is at least one of a barycenter of a face region, a barycenter of an eye region, a barycenter of a mouth region, a nostril barycentric position, a nose edge, and a barycenter of a flesh color region.

8. The apparatus according to claim 1, wherein the feature point pixel is at least one of a nostril barycenter, a barycenter of a nose edge, an end of an eyebrow, upper and lower feature point pixels of the eyebrow, a median of the eyebrow, an end of an eye, upper and lower feature point pixels of the eye, an end of a mouth, and upper and lower feature point pixels of the mouth.

9. The apparatus according to claim 1, wherein the feature point pixel is part of at least one of a shape of an eyebrow, a distance between a left eyebrow and a right eyebrow, a distance between an eyebrow and an eye, a distance between an eye end and a mouth end, a distance between a nostril barycenter and the mouth end, a distance between a nose edge and the mouth end, a distance between a barycenter of a face region and the mouth end, a distance between eye ends, a distance between upper and lower edges of an eye region, a distance between mouth ends, a distance between upper and lower edges of a mouth region, wrinkles in forehead and glabella regions, and wrinkles in left and right cheek regions.

10. The apparatus according to claim 1, wherein the feature point pixel is expressed by relative coordinates to the specific part.

11. The apparatus according to claim 1, wherein said determination device corrects a position of the feature point pixel on the basis of a layout of one of the feature point pixel and the specific part in the input image data and executes the determination on the basis of the feature point pixel with the corrected position.

12. The apparatus according to claim 1, further comprising a holding device adapted to hold the input image data when said determination device determines predetermined facial expression.

13. The apparatus according to claim 1, further comprising a display device adapted to display the input image data when said determination device determines predetermined facial expression.

14. The apparatus according to claim 1, further comprising a setting device adapted to set a region in the image data on the basis of the position of the specific part detected by said first detection device,

wherein said second detection device detects the feature point pixel of the face from the set region.

16. A non-transitory computer-readable storage medium which stores the computer program to cause a computer to function as the information processing apparatus of claim 1.

15. A control method stored in a storage medium for causing an information processing apparatus to execute processing of image data containing a face, comprising steps of:

inputting image data containing a face;

storing image data in a memory;

detecting, by a first detection device which has been learned by supervisory data with various kinds of variations, from the image data stored in the memory, positions of a plurality of specific parts of the face;

setting respective feature point pixel extraction areas for a plurality of feature point pixels of the specific parts in the image data on the basis of the detected positions of the specific parts;

detecting by a second detection device which has been learned by supervisory data with specific variation that is one of the various kinds of variations, the plurality of feature point pixels of the specific parts from the respective feature point pixel extraction areas; and

determining facial expression of the face on the basis of the detected feature point pixels,

wherein the second detection step has higher detection accuracy than detection accuracy of the first detection step, and the first detection step is robust to a variation in a detection target.