Pose-robust recognition

• US 9,323,980 B2
• Filed: 05/13/2011
• Issued: 04/26/2016
• Est. Priority Date: 05/13/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sun, Jian, Yin, Qi, Tang, Xiaoou
• Primary Examiner(s): Rush, Eric
• Application Number:
  US13/107,856
• Publication Number:
  US 20120288167A1
• Time in Patent Office: 1,810 Days
• Field of Search: 382115-118, 382/155, 382/156, 382/159, 382/160, 382/173, 382/181, 382/190, 382/195, 382/209, 382/224, 382/227, 382/282, 382294-298
• US Class Current: 1/1
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Some implementations provide techniques and arrangements to address intrapersonal variations encountered during facial recognition. For example, some implementations transform at least a portion of an image from a first intrapersonal condition to a second intrapersonal condition to enable more accurate comparison with another image. Some implementations may determine a pose category of an input image and may modify at least a portion of the input image to a different pose category of another image for comparing the input image with the other image. Further, some implementations provide for compression of data representing at least a portion of the input image to decrease the dimensionality of the data.

Claims

1. A method comprising:

under control of one or more processors configured with executable instructions,

determining a first pose category of at least a portion of a first image from among a plurality of pose categories;

transforming at least the portion of the first image to a second pose category corresponding to a second image to obtain a transformed image, wherein at least a portion of the transformed image is divided into a plurality of cells;

using multiple descriptors to extract features for each cell of the portion of the transformed image;

rescaling the results of the multiple descriptors determined for each cell;

concatenating the rescaled results to obtain a combined feature vector for each cell of the plurality of cells;

compressing the combined feature vectors to obtain a compact final descriptor representative of at least the portion of the transformed image; and

comparing the transformed image with the second image to determine, at least in part, whether the first image is a match with the second image.

2. The method as recited in claim 1, the transforming at least the portion of the first image to the second pose category further comprising applying geometric rectification to at least the portion of the first image based, at least in part, on an a scale factor determined from training data.

3. The method as recited in claim 2, wherein the training data comprises a plurality of sets of images including different pose categories of the plurality of pose categories, the method further comprising optimizing the scale factor based, at least in part, on the training data.

4. The method as recited in claim 2, further comprising performing local perturbation to refine a location of at least the portion of the first image and the scale factor based on pose categories and landmarks determined from the training data.

5. The method as recited in claim 1, the transforming at least the portion of the first image to the second pose category further comprising:

determining that at least a portion of a face in the first image is at least partially occluded;

selecting a symmetrical portion of the face the from an opposite side of the face;

flipping the symmetrical portion of the face; and

including in the transformed image the flipped symmetrical portion of the face in substitution for at least the portion that is at least partially occluded.

6. The method as recited in claim 1, the determining the first pose category further comprising:

generating based, at least in part, on training data a plurality of pose category templates, each pose category template representing a different pose category; and

comparing the first image with the plurality of pose category templates to identify a pose category for the first image based on similarity to one of the pose category templates.

7. The method as recited in claim 1, the method further comprising:

dividing the first image into a plurality of facial components;

for each facial component performing the transforming to transform the facial component from the first pose category to the second pose category; and

the comparing further comprising comparing a final descriptor determined for each facial component with a corresponding facial component from the second image.

8. The method as recited in claim 1, wherein:

the compressing comprises compressing the combined feature vector for each cell using standard principal component analysis (PCA) compression to obtain a compact descriptor result for each cell.

9. The method as recited in claim 8, wherein the combined feature vector for each cell represents a first combined feature vector, and the compressing further comprising:

concatenating the compact descriptor result for each cell to obtain a second combined feature vector representative of at least the portion of the transformed image; and

compressing the second combined feature vector using a variant PCA compression to obtain the compact final descriptor representative of at least the portion of the transformed image, wherein the variant PCA compression includes a whitening operation and a normalizing operation.

10. The method as recited in claim 1, wherein the results of the multiple descriptors determined for each cell are codes.

11. The method as recited in claim 1, further comprising rescaling the results of the multiple descriptors determined for each cell with an associated code number prior to concatenation.

12. Computer-readable storage media maintaining instructions executable by one or more processors to perform operations comprising:

receiving an image;

dividing a face in the image into a plurality of facial components;

using multiple descriptors to extract features from a facial component of the plurality of facial components;

rescaling the features extracted using the multiple descriptors;

concatenating the results of the rescaling to obtain a combined feature vector representative of the facial component; and

compressing results of the concatenating to obtain a compact final descriptor representative of the facial component.

13. The computer-readable storage media as recited in claim 12, the image comprising a first image, and the operations further comprising:

transforming the facial component of the plurality of facial components from a first pose category to a second pose category;

comparing the compact final descriptor representative of the facial component of the face in the image with a corresponding analogous facial component of a second image corresponding to the second pose category to determine, at least in part, whether there is a match between the first image and the second image.

14. The computer-readable storage media as recited in claim 13, the operations further comprising:

prior to the transforming, identifying one or more facial landmarks in the face in the first image; and

determining that the face in the first image corresponds to the first pose category based, at least in part on the facial landmarks.

15. The computer-readable storage media as recited in claim 14, the transforming comprising geometric rectification of the facial component based on a global affine transformation to bring the face into a position corresponding to the second pose category.

16. The computer-readable storage media as recited in claim 15, the transforming including application of a scale factor for performing geometric rectification, the operations further comprising optimizing the scale factor using training data including a plurality of identities, each identity including images of an individual in different pose categories.

17. A computing device comprising:

one or more processors in operable communication with computer-readable media;

a feature extraction module maintained on the computer-readable media and executed on the one or more processors to use multiple descriptors to extract features from a facial component of an input image;

a compression module maintained on the computer-readable media and executed on the one or more processors to compress the features extracted using the multiple descriptors to obtain a final descriptor representative of the facial component; and

a feature combination module maintained on the computer-readable media and executed on the one or more processors to rescale the features extracted using the multiple descriptors with an associated code number and concatenate the features extracted using the multiple descriptors to obtain a combined feature vector representative of at least a portion of the facial component.

18. The computing device as recited in claim 17, wherein the compression module is configured to apply a two-layer compression that compresses features extracted from a plurality of cells of the facial component using a first compression technique to obtain a descriptor representative of each cell, and that compresses a concatenation of the descriptors representative of each cell to obtain a final descriptor representative of the facial component.

19. The computing device as recited in claim 18, wherein a first layer of the two-layer compression is a standard principal component analysis (PCA) compression and the second layer of the two-layer compression incorporates at least one of a whitening operation or a normalizing operation into PCA compression.

20. The computing device as recited in claim 17, further comprising:

a pose estimation module maintained on the computer-readable media and executed on the one or more processors to receive the input image and determine a first pose category for the input image from among a plurality pose categories; and

a pose adaption module maintained on the computer-readable media and executed on the one or more processors to transform the facial component of the input image from the first pose category to a different pose category of the plurality of pose categories.