Face annotation for photo management

• US 20040264780A1
• Filed: 06/30/2003
• Published: 12/30/2004
• Est. Priority Date: 06/30/2003
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Li, Mingjing, Chen, Longbin, Zhang, Lei, Zhang, Hong-Jiang
• Granted Patent: US 7,274,822 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

Systems and methods for annotating a face in a digital image are described. In one aspect, a probability model is trained by mapping one or more sets of sample facial features to corresponding names of individuals. A face from an input data set of at least one the digital image is then detected. Facial features are then automatically extracted from the detected face. A similarity measure is them modeled as a posterior probability that the facial features match a particular set of features identified in the probability model. The similarity measure is statistically learned. A name is then inferred as a function of the similarity measure. The face is then annotated with the name.

Claims

1. A method comprising:

deriving a similarity measure by integrating results from content-based image retrieval and face recognition into a Bayesian framework to obtain a maximum posterior estimation that extended face area contextual features, face appearance, and face components associated with a set of facial features match a particular set of features identified in a probability model, the probability model mapping one or more sets of sample facial features independent of any specific individual to corresponding names of individuals; and

inferring a name of a particular person that corresponds to the facial features as a function of the similarity measure.

2. A method as recited in claim 1, wherein modeling the similarity measure further comprises marginalizing the similarity measure over one or more missing features to arrive at a similarity measure for the one or more missing features.

3. A method as recited in claim 1:

wherein the digital image is a first digital image; and

wherein deriving the similarity measure further comprises comparing the extended face area contextual features with contextual features extracted from a second digital images only if the first and second digital images were generated within a configurable number of days from one another, the second digital image being from a training data set.

4. A method as recited in claim 1, wherein inferring the name further comprises generating a candidate name list by integrating priors of both intra-personal variations Ω

₁ and inter-personal variations Ω

_E into the similarity measure.

5. A method as recited in claim 1:

wherein inferring the name further comprises searching historical labeling results to generate a candidate name list; and

wherein the method further comprises automatically annotating the face with the name if the similarity measure meets or exceeds a confidence threshold.

6. A method as recited in claim 1, wherein the face is an unknown face, and wherein inferring the name further comprises:

integrating priors of both intra-personal variations Ω

_E and inter-personal variations Ω

_E into the similarity measure to generate a candidate name list, the candidate name list comprising the name;

based on a nearest neighbor or K-nearest neighbor algorithm, among K nearest labeled faces to the unknown face, sorting names in the candidate name list according to a sum of similarities measures corresponding to the unknown face; and

presenting the candidate list to a user for annotation.

7. A method as recited in claim 1, wherein inferring the name further comprises:

obtaining labeling result from a user;

storing the labeling result into a historical record;

performing multi-class classifications to identify a candidate name list as a function of the historical record; and

wherein annotating the face further comprises presenting the candidate name list to a user, the name being in the candidate name list.

8. A computer-readable medium comprising computer-program instructions executable by a processor for performing a method as recited in claim 1.

9. A computer-readable medium comprising computer-program instructions executable by a processor, the computer-program instructions comprising instructions for:

training a probability model to map one or more sets of sample facial features to corresponding names of individuals;

detecting a face in the digital image;

extracting facial features from the face;

modeling a similarity measure as a posterior probability that the facial features match a particular set of features identified in the probability model, the similarity measure being statistically learned;

inferring a name as a function of the similarity measure; and

annotating the face with the name.

10. A computer-readable medium as recited in claim 9, wherein the digital image is from a family photo album.

11. A computer-readable medium as recited in claim 9, wherein the facial features comprise face appearance, face component, and extended face area contextual features.

12. A computer-readable medium as recited in claim 9, wherein the facial features comprise face appearance, face component, and extended face area contextual features, the face appearance and components being modeled in eigenspaces.

13. A computer-readable medium as recited in claim 9, wherein the instructions for modeling the similarity measure further comprise computer-program instructions for integrating results from content-based image retrieval and face recognition algorithms into a Bayesian framework.

14. A computer-readable medium as recited in claim 9, wherein the instructions for modeling the similarity measure further comprise computer-program instructions for marginalizing the similarity measure over one or more missing features to arrive at a similarity measure for the one or more missing features.

15. A computer-readable medium as recited in claim 9, wherein the facial features comprise contextual features, and wherein the instructions for modeling the similarity measure further comprise computer-program instructions for comparing contextual features of the face with contextual features from other digital images only if the other digital images were generated within a configurable number of days from when the digital image was created.

16. A computer-readable medium as recited in claim 9, wherein the instructions for inferring the name further comprise computer-program instructions for searching historical labeling results to generate a candidate name list.

17. A computer-readable medium as recited in claim 9, wherein the instructions for inferring the name further comprise computer-program instructions for generating a candidate name list by integrating priors of both intra-personal variations Ω

₁ and inter-personal variations Ω

_E into the similarity measure.

18. A computer-readable medium as recited in claim 9, wherein the face is an unknown face, and wherein the instructions for inferring the name further comprise computer-program instructions for:

integrating priors of both intra-personal variations Ω

₁ and inter-personal variations Ω

_E into the similarity measure to generate a candidate name list, the candidate name list comprising the name;

based on a nearest neighbor or K-nearest neighbor algorithm, among K nearest labeled faces to the unknown face, sorting names in the candidate name list according to a sum of similarities measures corresponding to the unknown face; and

presenting the candidate list to a user for annotation.

19. A computer-readable medium as recited in claim 9, wherein the instructions for inferring the name further comprise computer-program instructions for:

obtaining labeling result from a user;

storing the labeling result into a historical record;

performing multi-class classifications to identify a candidate name list as a function of the historical record; and

wherein annotating the face further comprises presenting the candidate name list to a user, the name being in the candidate name list.

20. A computer-readable medium as recited in claim 9, further comprising computer-program instructions for presenting the name in a user interface (UI), the UI allowing a user to accept or reject the name.

21. A computing device comprising a processor and a memory, the memory comprising computer-program instructions executable by the processor, the computer-program instructions comprising instructions for:

extracting facial features from a detected face in digital image data;

deriving a similarity measure by integrating results from content-based image retrieval and face recognition into a Bayesian framework to obtain a maximum posterior estimation that extended face area contextual features, face appearance, and face components associated with the facial features match a particular set of features identified in a probability model, the probability model mapping one or more sets of sample facial features independent of any specific individual to corresponding names of individuals; and

inferring a name as a function of the similarity measure.

22. A computing device as recited in claim 21, wherein the instructions for modeling the similarity measure further comprise computer-program instructions for marginalizing the similarity measure over one or more missing features to arrive at a similarity measure for the one or more missing features.

23. A computing device as recited in claim 21, wherein the digital image is a first digital image, and wherein the instructions for deriving the similarity measure further comprise computer-program instructions for comparing the extended face area contextual features with contextual features extracted from a second digital images only if the first and second digital images were generated within a configurable number of days from one another, the second digital image being from a training data set.

24. A computing device as recited in claim 21, wherein the instructions for inferring the name further comprise computer-program instructions for generating a candidate name list by integrating priors of both intra-personal variations Ω

₁ and inter-personal variations Ω

_E into the similarity measure.

25. A computing device as recited in claim 21:

wherein the instructions for inferring the name further comprise computer-program instructions for searching historical labeling results to generate a candidate name list; and

wherein the computer program instructions further comprise automatically annotating the face with the name if the similarity measure meets or exceeds a confidence threshold.

26. A computing device as recited in claim 21, wherein the face is an unknown face, and wherein the instructions for inferring the name further comprise computer-program instructions for:

integrating priors of both intra-personal variations Ω

₁ and inter-personal variations Ω

_E into the similarity measure to generate a candidate name list, the candidate name list comprising the name;

based on a nearest neighbor or K-nearest neighbor algorithm, among K nearest labeled faces to the unknown face, sorting names in the candidate name list according to a sum of similarities measures corresponding to the unknown face; and

presenting the candidate list to a user for annotation.

27. A computing device as recited in claim 21, wherein the instructions for inferring the name further comprise computer-program instructions for:

obtaining labeling result from a user;

storing the labeling result into a historical record;

performing multi-class classifications to identify a candidate name list as a function of the historical record; and

wherein annotating the face further comprises presenting the candidate name list to a user, the name being in the candidate name list.

28. A computing device as recited in claim 21, wherein the computer-program instructions further comprise instructions for annotating the face with the name.

29. A computing device comprising:

means for deriving a similarity measure by integrating results from content-based image retrieval and face recognition into a Bayesian framework to obtain a maximum posterior estimation that extended face area contextual features, face appearance, and face components associated with a set of facial features match a particular set of features identified in a probability model, the probability model mapping one or more sets of sample facial features independent of any specific individual to corresponding names of individuals; and

means for inferring a name of a particular person that corresponds to the facial features as a function of the similarity measure.

30. A computing device as recited in claim 29, wherein the facial features are associated with a face extracted from a digital photograph.

31. A computing device as recited in claim 29, wherein the means for modeling the similarity measure further comprise means for marginalizing the similarity measure over one or more missing features to arrive at a similarity measure for the one or more missing features.

32. A computing device as recited in claim 29:

wherein the digital image is a first digital image; and

wherein the means for deriving the similarity measure further comprise means for comparing the extended face area contextual features with contextual features extracted from a second digital images only if the first and second digital images were generated within a configurable number of days from one another, the second digital image being from a training data set.

33. A computing device as recited in claim 29, wherein the means for inferring the name further comprise means for generating a candidate name list by integrating priors of both intra-personal variations Ω

₁ and inter-personal variations Ω

_E into the similarity measure.

34. A computing device as recited in claim 29:

wherein the means for inferring the name further comprise means for searching historical labeling results to generate a candidate name list; and

wherein the computing device further comprise means for automatically annotating the face with the name if the similarity measure meets or exceeds a confidence threshold.

35. A computing device as recited in claim 29, wherein the face is an unknown face, and wherein the means for inferring the name further comprise:

means for integrating priors of both intra-personal variations Ω

₁ and inter-personal variations Ω

_E into the similarity measure to generate a candidate name list, the candidate name list comprising the name;

means for based on a nearest neighbor or K-nearest neighbor algorithm, among K nearest labeled faces to the unknown face, sorting names in the candidate name list according to a sum of similarities measures corresponding to the unknown face; and

means for presenting the candidate list to a user for annotation.

36. A computing device as recited in claim 29, wherein the means for inferring the name further comprises:

means for obtaining labeling result from a user;

means for storing the labeling result into a historical record;

means for performing multi-class classifications to identify a candidate name list as a function of the historical record; and

means for wherein annotating the face further comprises presenting the candidate name list to a user, the name being in the candidate name list.