ASSOCIATION AND PREDICTION IN FACIAL RECOGNITION

• US 20120288166A1
• Filed: 05/13/2011
• Published: 11/15/2012
• Est. Priority Date: 05/13/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sun, Jian, Yin, Qi, Tang, Xiaoou
• Granted Patent: US 9,251,402 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Some implementations provide techniques and arrangements to address intrapersonal variations encountered during facial recognition. For example, some implementations employ an identity data set having a plurality of images representing different intrapersonal settings. A predictive model may associate one or more input images with one or more images in the identity data set. Some implementations may use an appearance-prediction approach to compare two images by predicting an appearance of at least one of the images under an intrapersonal setting of the other image. Further, some implementations may utilize a likelihood-prediction approach for comparing images that generates a classifier for an input image based on an association of an input image with the identity data set.

Claims

1. A method comprising:

under control of one or more processors configured with executable instructions,

determining a first intrapersonal setting of at least a portion of a first image;

determining a second intrapersonal setting of at least a portion of a second image;

determining, based at least in part on a threshold, whether the first intrapersonal setting and the second intrapersonal setting are comparable; and

performing direct appearance matching between the first image and the second image when the first intrapersonal setting and the second intrapersonal setting are determined to be comparable.

2. The method as recited in claim 1, the determining the first intrapersonal setting of the first image further comprising:

accessing an identity data set including one or more identities, each identity in the identity data set including multiple images having different intrapersonal settings; and

associating at least the portion of the first image with at least one identity in the identity data set.

3. The method as recited in claim 2, the associating at least the portion of the first image with at least one identity in the identity data set further comprising:

dividing the first image into a plurality of facial components; and

for each facial component;

comparing a selected facial component from the first image with corresponding facial components contained in the multiple images in the one or more identities in the identity data set;

identifying, based on at least one descriptor distance, a facial component in the identity data set that is a closest match to the selected facial component from the first image; and

determining, as the first intrapersonal setting, an intrapersonal setting of the facial component in the identity data set that is the closest match to the selected facial component from the first image.

4. The method as recited in claim 2, the accessing the identity data set further comprising accessing a plurality of identities, each identity containing a plurality of images of different people in a variety of intrapersonal settings comprising at least one of:

different poses;

different lighting;

different facial expressions;

or

different obscured facial features.

5. The method as recited in claim 1, the determining whether the first intrapersonal setting and the second intrapersonal setting are comparable further comprising:

accessing an identity data set including multiple identities, each identity in the identity data set including multiple images categorized in different categories of intrapersonal settings;

determining an intrapersonal setting first category value corresponding to at least the portion of the first image;

determining an intrapersonal setting category second value corresponding to at least the portion of the second image; and

determining that the first intrapersonal setting and the second intrapersonal setting are comparable when a difference between the first category value and the second category value is not within the threshold.

6. The method as recited in claim 1, further comprising performing prediction-based matching between the first image and the second image when the first intrapersonal setting and the second intrapersonal setting are determined to not be comparable.

7. The method as recited in claim 6, the performing the prediction-based matching comprising performing at least one of:

appearance-prediction matching based on comparing a predicted portion of the first image, predicted for the second intrapersonal setting, with the portion of the second image;

or

likelihood-prediction matching using a classifier generated for the first image based on similarity to one or more identities in an identity data set.

8. The method as recited in claim 7, the performing the appearance-prediction matching further comprising comparing a predicted portion of the second image, predicted for the first intrapersonal setting, with the portion of the first image.

9. The method as recited in claim 7, the performing the likelihood-prediction matching further comprising generating the classifier by selecting a first subset of identities in the identity data set as negative samples and selecting at least one identity in the identity data set as a positive sample, the at least one identity determined to be similar to at least the portion of the first image.

10. The method as recited in claim 9, further comprising feeding at least the portion of the second image into the classifier to determine a likelihood of at least the portion of the second image being a match with at least the portion of the first image.

11. Computer-readable media maintaining instructions executable by one or more processors to perform operations comprising:

receiving a first image and a second image for comparison;

associating a facial component of the first image with a first identity of an identity data set, the identity data set including a plurality of identities, each identity including a plurality of images related to different intrapersonal settings;

associating a facial component of the second image with a second identity of the identity data set;

determining an intrapersonal setting for the facial component of the first image based on the first identity;

determining an intrapersonal setting for the facial component of the second image based on the second identity;

selecting, from the first identity, a particular image corresponding to the intrapersonal setting for the facial component of the second image; and

comparing a facial component of the particular image from the first identity with the facial component of the second image to determine, at least in part, whether there is a match between the second image and the first image.

12. The computer-readable media as recited in claim 11, the operations further comprising:

selecting, from the second identity, a particular image corresponding to the intrapersonal setting for the facial component of the first image; and

comparing a facial component of the particular image from the second identity with the facial component of the first image to determine, at least in part, whether there is a match between the second image and the first image.

13. The computer-readable media as recited in claim 12, the operations further comprising:

obtaining a first result from the comparing the facial component of the particular image from the first identity with the facial component of the second image;

obtaining a second result from the comparing the facial component of the particular image from the second identity with the facial component of the first image; and

combining the first result with the second result to obtain a final result as to whether the facial component of the first image matches the facial component of the second image.

14. The computer-readable media as recited in claim 13, the operations further comprising:

obtaining a plurality of final results for a plurality of different facial components of the first image and the second image; and

combining the plurality of final results to determine, at least in part, whether there is a match between the first image and the second image.

15. The computer-readable media as recited in claim 11, the operations further comprising:

prior to the selecting, establishing that the first intrapersonal setting is in a first category of settings and that the second intrapersonal setting is in a second category of settings;

calculating whether a distance between the first category and the second category is within a threshold; and

determining, based at least in part on the threshold, that the first intrapersonal setting and the second intrapersonal setting are not comparable.

16. A computing device comprising:

one or more processors in operable communication with computer-readable media;

an identity data set stored on the computer-readable media, the identity data set including a plurality of identities, each identity including a plurality of images set in different intrapersonal settings;

an association component maintained on the computer-readable media and executed on the one or more processors to associate at least a portion of a first image with a subset of identities in the identity data set; and

a likelihood-prediction component maintained on the computer-readable media and executed on the one or more processors to;

generate a classifier based on the association of at least the portion of the first image with the subset of identities; and

feed at least a portion of a second image into the classifier to determine, at least in part, whether there is a match between the first image and the second image.

17. The computing device as recited in claim 16, wherein the association component is further executed to associate at least the portion of the first image with the subset of identities based on a similarity between at least the portion of the first image and images in the subset of identities, the similarity being determined based on one or more descriptors.

18. The computing device as recited in claim 17, wherein the classifier includes a plurality of positive image samples taken from the subset of identities and a plurality of negative image samples taken from other identities than those in the subset of identities.

19. The computing device as recited in claim 16, further comprising:

a switching mechanism maintained on the computer-readable media and executed on the one or more processors to determine, based at least in part on a threshold, a difference between a first intrapersonal setting for at least the portion of the first image and a second intrapersonal setting for at least the portion of the second image; and

a direct appearance-matching component maintained on the computer-readable media and executed on the one or more processors to perform direct appearance-based matching between the first image and the second image when the switching mechanism determines that the first intrapersonal setting is comparable to the second intrapersonal setting.

20. The computing device as recited in claim 16, further comprising an appearance-prediction component maintained on the computer-readable media and executed on the one or more processors to:

select, from one of the identities of the subset of identities associated with at least the portion of the first image, a particular image corresponding to an intrapersonal setting for at least the portion of the second image; and

compare at least a portion of the particular image from the first identity with the facial component of the second image to determine, at least in part, whether there is a match between the second image and the first image