Apparatus, method, and program for discriminating subjects

• US 7,920,725 B2
• Filed: 09/08/2004
• Issued: 04/05/2011
• Est. Priority Date: 09/09/2003
• Status: Expired due to Fees

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Li, Yuanzhong
• Primary Examiner(s): Chen; Wenpeng
• Assistant Examiner(s): Thirugnanam; Gandhi
• Application Number:
  US10/935,336
• Publication Number:
  US 20050100195A1
• Time in Patent Office: 2,400 Days
• Field of Search: 382/118, 382/173
• US Class Current: 382/118
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

A characteristic amount calculating means calculates first characteristic amounts, which do not require normalization, and normalized second characteristic amounts. A first discriminating portion discriminates whether a candidate for a face is included in the target image, by referring to first reference data with the first characteristic amounts, calculated from the target image. The first reference data is obtained by learning the first characteristic amounts of a plurality of images, which are known either to be of faces or to not be of faces. In the case that the candidate is included, a second discriminating portion discriminates whether the candidate is a face, by referring to second reference data, obtained by learning the second characteristic amounts of a plurality of images, which known either to be of faces or to not to be of faces.

Claims

1. A subject discriminating apparatus comprising:

an image input means for receiving input of a target image, which is a target of discrimination;

a first characteristic amount calculating means for calculating first characteristic amounts including directions of gradient vectors or color data, which do not require normalization and are employed to discriminate a predetermined subject, from the target image which is being currently received;

a first discriminating means for discriminating whether a candidate of the predetermined subject is included in the target image based on the first characteristic amounts, which are calculated from the target image only, by referring to first reference data, in which learned first characteristic amounts and first discrimination conditions corresponding to the learned first characteristic amounts are defined in advance from sample images of various subjects;

a second characteristic amount calculating means for calculating second characteristic amounts, which are normalized and employed to discriminate the predetermined subject, from the candidate of the predetermined subject only when the first discriminating means judges that the candidate is included in the target image;

a second discriminating means for discriminating whether the candidate is the predetermined subject based on the normalized second characteristic amounts, by referring to second reference data, in which learned second characteristic amounts and second discrimination conditions corresponding to the learned second characteristic amounts are defined in advance from the sample images; and

a control means, for comparing a number of times N1 that the first discriminating means judges that a candidate is included in the target image to a number of times N2 that the second discriminating means judges that the candidate is a face, judging whether a ratio of the number of times N2 to the number of times N1 is equal to or greater than a predetermined percentage when a number of target images for which discriminating has been performed is equal to a predetermined number, and if the judgment is affirmative, controlling the first characteristic amount calculating means, the first discriminating means, the second characteristic amount calculating means, and the second discriminating means so that the candidate discriminated by the first discriminating means is judged to be the predetermined subject by calculating only the first characteristic amounts from the target images which are subsequently discriminated.

2. The subject discriminating apparatus as defined in claim 1, wherein:

the first discriminating means obtains the first reference data in advance, by learning the learned first characteristic amounts within a plurality of sample images, which are known to include the predetermined subject, and a plurality of sample image groups, which are known to not include the predetermined subject, with a machine learning technique.

3. The subject discriminating apparatus as defined in claim 2, wherein:

if the predetermined subject is a face, the first discriminating means obtains the first reference data by learning the learned first characteristic amounts within a first region that includes a left eye and left cheek and within a second region that includes a right eye and right cheek of the sample images, which are known to include the predetermined subject, and by learning the learned first characteristic amounts within regions corresponding to the first and second regions of the sample images, which are known to not include the predetermined subject; and

the first characteristic amount calculating means calculates the first characteristic amounts of regions that correspond to the first and second regions within the target image.

4. The subject discriminating apparatus as defined in claim 3, wherein:

the first discriminating means obtains the first reference data, by learning the learned first characteristic amounts within a third region that includes the left and right eyes of the sample images, which are known to include the predetermined subject and by learning the learned first characteristic amounts within regions corresponding to the third region of the sample images, which are known to not include the predetermined subject; and

the first characteristic amount calculating means calculates the first characteristic amounts of regions that correspond to the first, second, and third regions within the target image.

5. The subject discriminating apparatus as defined in claim 1, wherein:

the second discriminating means obtains the second reference data in advance by learning the learned second characteristic amounts within a plurality of sample images, which are known to include the predetermined subject, and a plurality of sample image groups, which are known to not include the predetermined subject, with a machine learning technique.

6. The subject discriminating apparatus as defined in claim 5, wherein:

if the predetermined subject is a face, the second discriminating means obtains the second reference data by learning the learned second characteristic amounts within a first region that includes a left eye and left cheek and within a second region that includes a right eye and right cheek of the sample images, which are known to include the predetermined subject, and by learning the learned second characteristic amounts within regions corresponding to the first and second regions of the sample images, which are known to not include the predetermined subject; and

the second characteristic amount calculating means calculates the second characteristic amounts of regions that correspond to the first and second regions within the target image.

7. The subject discriminating apparatus as defined in claim 6, wherein:

the second discriminating means obtains the second reference data by learning the learned second characteristic amounts within a third region that includes the left and right eyes of the sample images, which are known to include the predetermined subject, and by learning the learned second characteristic amounts within regions corresponding to the third region of the sample images, which are known to not include the predetermined subject; and

the second characteristic amount calculating means calculates the second characteristic amounts of regions that correspond to the first, second, and third regions within the target image.

8. The subject discriminating apparatus as defined in claim 1, wherein:

the second characteristic amounts are directions and magnitudes of gradient vectors of each pixel of the received image.

9. The subject discriminating apparatus as defined in claim 1, further comprising:

at least one other discriminating means for discriminating whether the predetermined subject, judged to be included in the target image by the second discriminating means, is the predetermined subject, based on other characteristic amounts, which are calculated from the candidate.

10. The subject discriminating apparatus as defined in claim 1, further comprising:

an extracting means for extracting the predetermined subject from the target image.

11. The subject discriminating apparatus as defined in claim 1, further comprising:

an output means for attaching data that represents a position of the predetermined subject within the target image, and outputting the data with the target image.

12. A photography apparatus equipped with the subject discriminating apparatus defined in claim 1.

13. The photography apparatus as defined in claim 12, which photography apparatus takes frontal images.

14. The subject discriminating apparatus as defined in claim 1, wherein the first characteristic amounts comprise only directions of gradient vectors of each pixel of the target image.

15. The subject discriminating apparatus as defined in claim 14, wherein the second characteristic amounts comprise the direction and a magnitude of the gradient vector of each pixel of the target image.

16. The subject discriminating apparatus as defined in claim 1, wherein the first discriminating means judges that the candidate of the predetermined subject is not included in the target image based on the calculated first characteristic amounts and the first reference data.

17. The subject discriminating apparatus as defined in claim 16, further comprising:

an output means for outputting discrimination results indicating that the candidate of the predetermined subject is not included in the target image based on a judgment of the first discriminating means.

18. The subject discriminating apparatus as defined in claim 16, wherein the first discriminating means identifies candidate faces in the target image.

19. The subject discriminating apparatus as defined in claim 16, wherein the first discrimination conditions are defined for combinations of the learned first characteristic amounts.

20. The subject discriminating apparatus as defined in claim 1, wherein the second characteristic amounts comprise magnitudes of gradient vectors of each pixel of the candidate, which magnitudes are normalized by flattening a histogram of the magnitudes so that a distribution of the magnitudes is evenly distributed across a predetermined range of values for the pixels of the candidate.

21. The subject discriminating apparatus as defined in claim 1, wherein the first characteristic amounts comprise parameters which do not change dependent on changes in brightness or contrast of the image.

22. The subject discriminating apparatus as defined in claim 1, wherein the calculated first characteristic amounts comprise the directions of the gradient vectors of pixels of the received target image and the first reference data comprises directions of the gradient vectors determined from the sample images and respective first discrimination conditions.

23. The subject discriminating apparatus as defined in claim 22, wherein the first discriminating means discriminates the candidate of the predetermined subject from the target image by comparing the calculated direction of the gradient vector of each pixel to the first reference data.

24. The subject discriminating apparatus as defined in claim 23, wherein the second characteristic amount comprises values of a magnitude of the gradient vector of each pixel of the discriminated candidate and the second reference data comprises magnitudes and directions of the gradient vectors determined from the sample images and respective second discrimination conditions.

25. The subject discriminating apparatus as defined in claim 24, wherein the second discriminating means discriminates the predetermined subject from the candidate by comparing the magnitudes and directions of the gradient vectors of each pixel to the second reference data.

26. The subject discriminating apparatus as defined in claim 1, wherein the second characteristic amounts comprise values of a magnitude of the gradient vector of each pixel of the candidate discriminated by the first discriminating means and the second reference data comprises magnitudes and directions of the gradient vectors and respective second discrimination conditions.

27. A subject discriminating method comprising:

receiving input of a target image, which is a target of discrimination;

calculating first characteristic amounts including directions of gradient vectors or color data, which do not require normalization and are employed to discriminate a predetermined subject, from the target image which is being currently received;

first discriminating whether a candidate of the predetermined subject is included in the target image based on the first characteristic amounts, which are calculated from the target image only, by referring to first reference data, in which learned first characteristic amounts and discrimination conditions corresponding to the learned first characteristic amounts are defined in advance from sample images of various subjects;

calculating second characteristic amounts, which are normalized and employed to discriminate the predetermined subject, from the candidate of the predetermined subject if the first discriminating means judges that the candidate is included in the target image;

second discriminating whether the candidate is the predetermined subject based on the normalized second characteristic amounts, by referring to second reference data, in which learned second characteristic amounts and discrimination conditions corresponding to the learned second characteristic amounts are defined in advance from the sample images;

comparing a number of times N1 when it is judged in the first discriminating that a candidate is included in the target image to a number of times N2 when it is judged in the second discriminating that the candidate is a face and judging whether a ratio of the number of times N2 to the number of times N1 is equal to or greater than a predetermined percentage when a number of target images for which the first discriminating and the second discriminating have been performed is equal to a predetermined number; and

if the judgment is affirmative, performing controlling of the calculating the first characteristic amounts, the first discriminating, the calculating the second characteristic amounts, and the second discriminating so that the candidate discriminated in the first discriminating is judged to be the predetermined subject by calculating only the first characteristic amounts from the target images which are subsequently discriminated.

28. The method as defined in claim 27, wherein discriminating whether the candidate is the predetermined subject comprises:

discriminating whether the candidate is the predetermined subject based on the normalized second characteristic amounts, calculated first characteristic amounts, and the second reference data.

29. A non-transitory computer-readable memory storing a program that, when executed by a computer, causes the computer to execute a subject discriminating method, the method comprising:

receiving input of a target image, which is a target of discrimination;

calculating first characteristic amounts including directions of gradient vectors or color data, which do not require normalization and are employed to discriminate a predetermined subject, from the target image which is being currently received;

first discriminating whether a candidate of the predetermined subject is included in the target image based on the first characteristic amounts, which are calculated from the target image only, by referring to first reference data, in which learned first characteristic amounts and discrimination conditions corresponding to the learned first characteristic amounts are defined in advance from sample images of various subjects;

calculating second characteristic amounts, which are normalized and employed to discriminate the predetermined subject, from the candidate of the predetermined subject if the first discriminating means judges that the candidate is included in the target image;

second discriminating whether the candidate is the predetermined subject based on the normalized second characteristic amounts, by referring to second reference data, in which learned second characteristic amounts and discrimination conditions corresponding to the learned second characteristic amounts are defined in advance from the sample images;

comparing a number of times N1 that it is judged in the first discriminating that a candidate is included in the target image to a number of times N2 that is judged in the second discriminating that the candidate is a face and judging whether a ratio of the number of times N2 to the number of times N1 is equal to or greater than a predetermined percentage when a number of target images for which the first discriminating and the second discriminating have been performed is equal to a predetermined number; and

if the judgment is affirmative, performing controlling of the calculating the first characteristic amounts, the first discriminating, the calculating second characteristic amounts, and the second discriminating so that the candidate discriminated in the first discriminating is judged to be the predetermined subject by calculating only the first characteristic amounts from the target images which are subsequently discriminated.