Association and prediction in facial recognition

• US 9,251,402 B2
• Filed: 05/13/2011
• Issued: 02/02/2016
• Est. Priority Date: 05/13/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sun, Jian, Yin, Qi, Tang, Xiaoou
• Primary Examiner(s): Rush, Eric
• Application Number:
  US13/107,800
• Publication Number:
  US 20120288166A1
• Time in Patent Office: 1,726 Days
• Field of Search: 382115-118, 382/155, 382/156, 382/159, 382/160, 382/173, 382/181, 382/190, 382/195, 382/209, 382/224, 382/227, 382/282, 382/294
• US Class Current: 1/1
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Some implementations provide techniques and arrangements to address intrapersonal variations encountered during facial recognition. For example, some implementations employ an identity data set having a plurality of images representing different intrapersonal settings. A predictive model may associate one or more input images with one or more images in the identity data set. Some implementations may use an appearance-prediction approach to compare two images by predicting an appearance of at least one of the images under an intrapersonal setting of the other image. Further, some implementations may utilize a likelihood-prediction approach for comparing images that generates a classifier for an input image based on an association of an input image with the identity data set.

Claims

1. A method comprising:

under control of one or more processors configured with executable instructions, determining a first intrapersonal setting of at least a portion of a first input image;

determining a second intrapersonal setting of at least a portion of a second image;

determining, based at least in part on a threshold for a difference between the first intrapersonal setting and the second intrapersonal setting, whether the first input image and the second image are comparable with each other for direct appearance matching; and

in response to determining that the first input image and the second image are not comparable with each other for direct appearance matching;

identifying a plurality of images of an identity data set that are associated with a different identity than an identity associated with the first input image;

identifying a nearest match between the at least the portion of the first input image and at least a portion of a first generic image of the plurality of images, wherein the first generic image is associated with a generic identity;

transforming the first input image into a third image by substituting at least a portion of a second generic image of the plurality of images with the at least the portion of the first input image, wherein the second generic image is associated with the generic identity; and

performing direct appearance matching between the second image and the third image.

2. The method as recited in claim 1, the determining the first intrapersonal setting of the at least the portion of the first input image further comprising:

accessing the identity data set including one or more identities, each identity in the identity data set including multiple images having different intrapersonal settings; and

associating the at least the portion of the first input image with at least one identity in the identity data set.

3. The method as recited in claim 2, the associating the at least the portion of the first input image with at least one identity in the identity data set further comprising:

dividing the first input image into a plurality of facial components; and

for each facial component;

comparing a selected facial component from the first input image with corresponding facial components contained in the multiple images in the one or more identities in the identity data set;

identifying, based on at least one descriptor distance, a facial component in the identity data set that is a nearest component match to the selected facial component from the first input image; and

determining, as the first intrapersonal setting, an intrapersonal setting of the facial component in the identity data set that is the nearest component match to the selected facial component from the first input image, wherein the facial component of the identity data set is associated with a different identity than the selected facial component from the first input image.

4. The method as recited in claim 2, the accessing the identity data set further comprising accessing a plurality of identities, a first identity in the plurality of identity containing a plurality of images of the first identity in a variety of intrapersonal settings comprising at least one of:

different poses;

different lighting;

different facial expressions;

or

different obscured facial features.

5. The method as recited in claim 1, the determining whether the first input image and the second image are comparable further comprising:

accessing the identity data set including a plurality of identities, each identity of the plurality of identities including multiple images categorized in different categories of intrapersonal settings;

determining an intrapersonal setting first category value corresponding to the at least the portion of the first input image;

determining an intrapersonal setting second category value corresponding to the at least the portion of the second image; and

determining that the first input image and the second image are comparable when a difference between the first category value and the second category value is within the threshold for the difference between the first intrapersonal setting and the second intrapersonal setting.

6. The method as recited in claim 1, further comprising, in response to determining that the first input image and the second image are comparable with each other for direct appearance matching, performing direct appearance matching between the first input image and the second image.

7. The method as recited in claim 6, further comprising performing likelihood-prediction matching using a classifier generated for the first input image based on similarity of the first input image to one or more identities in the identity data set.

8. The method as recited in claim 7, further comprising generating the classifier by selecting a first subset of identities in the identity data set as negative samples and selecting at least one identity in the identity data set as a positive sample, the at least one identity determined to be most similar of identities in the identity data set to at least the portion of the first input image.

9. The method as recited in claim 8, further comprising feeding the at least the portion of the second image into the classifier to determine a likelihood of the at least the portion of the second image being a match with the at least the portion of the first input image.

10. The method as recited in claim 1, wherein the transforming is further based on comparing a predicted portion of the second image, predicted for the first intrapersonal setting, with the portion of the first input image.

11. One or more computer storage media storing instructions executable by one or more processors to perform operations comprising:

determining a first intrapersonal setting of at least a portion of a first input image;

determining a second intrapersonal setting of at least a portion of a second image; and

determining whether the first input image and the second image are comparable with each other for direct appearance matching based at least in part on a difference between the first intrapersonal setting and the second intrapersonal setting;

in response to determining that the first input image and the second image are not comparable with each other for direct appearance matching, transforming the first input image into a third image by substituting the at least the portion of the first input image with a portion of a generic image associated with a generic identity in an identity data set comprising a plurality of images.

12. The one or more computer storage media as recited in claim 11, the operations further comprising performing direct appearance matching between the first input image and the second image in response to determining that the first input image and the second image are comparable with each other for direct appearance matching.

13. The one or more computer storage media as recited in claim 11, the operations further comprising determining that one or more images in the plurality of images of the identity data set is similar in appearance to the second image.

14. The one or more computer storage media as recited in claim 11, wherein the transforming the first input image into the third image is based on one or more transitions between the plurality of images of the identity data set.

15. The one or more computer storage media as recited in claim 11, the operations further comprising performing direct appearance matching between the second image and the third image.

16. A system comprising:

one or more processors;

a memory that includes a plurality of processor-executable instructions to cause the one or more processors to perform operations comprising;

accessing an identity data set including one or more identities, each identity in the identity data set including multiple images having different intrapersonal settings;

associating at least a portion of a first input image with at least one identity in the identity data set;

determining a first intrapersonal setting of at least the portion of the first input image based on the associating;

determining a second intrapersonal setting of at least a portion of a second image;

determining whether to perform direct appearance matching or prediction-based matching between the first input image and the second image based on a difference between the first intrapersonal setting and the second intrapersonal setting;

in response to determining to perform prediction-based matching between the first input image and the second image, transforming the first input image into a third image based at least on transitions between images of the identity data set and at least one substitution of the at least the portion of the first input image with at least a portion of a generic image associated with the at least one identity in the identity data set, wherein the at least one identity is a generic identity; and

performing direct appearance matching between the third image and the second image.

17. The system as recited in claim 16, the associating at least the portion of the first input image with the at least one identity in the identity data set further comprising:

dividing the first input image into a plurality of facial components; and

for each facial component;

comparing a selected facial component from the first input image with corresponding facial components contained in the multiple images in the one or more identities in the identity data set;

identifying, based on at least one descriptor distance, a facial component in the identity data set that is a nearest match to the selected facial component from the first input image; and

determining, as the first intrapersonal setting, an intrapersonal setting of the facial component in the identity data set that is the nearest match to the selected facial component from the first input image, wherein the facial component of the identity data set is associated with a different identity than the selected facial component from the first input image.

18. The system as recited in claim 16, wherein the operations further comprise performing the prediction-based matching between the first input image and the second image based at least in part on determining that the difference between the first intrapersonal setting and the second intrapersonal setting is not within a threshold for the difference.

19. The system as recited in claim 18, the performing the prediction-based matching comprising performing likelihood-prediction matching using a classifier generated for the first input image based on similarity of the first input image to the one or more identities in the identity data set.

20. The system as recited in claim 19, the performing the likelihood-prediction matching further comprising generating the classifier by selecting a first subset of identities in the identity data set as negative samples and selecting at least one identity in the identity data set as a positive sample, the at least one identity determined to be most similar of identities in the identity data set to the at least the portion of the first input image.