Building a person profile database

• US 8,559,682 B2
• Filed: 11/09/2010
• Issued: 10/15/2013
• Est. Priority Date: 11/09/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Lei, Wang, Xin-Jing, Ma, Wei-Ying
• Primary Examiner(s): MARIAM, DANIEL G
• Application Number:
  US12/942,284
• Publication Number:
  US 20120114197A1
• Time in Patent Office: 1,071 Days
• Field of Search: 382/118, 382/159, 382/190, 382/209, 382/218, 382/224, 382/305
• US Class Current: 382/118
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions

Abstract

Names of entities, such as people, in an image may be identified automatically. Visually similar images of entities are retrieved, including text proximate to the visually similar images. The collected text is mined for names of entities, and the detected names are analyzed. A name may be associated with the entity in the image, based on the analysis.

Claims

1. A system for automatically identifying a name of a person in an image, the system comprising:

a processor;

memory coupled to the processor;

an analysis component stored in the memory and operable on the processor to;

receive a query including a face image,

detect visual features from the face image,

collect visually similar images based on the detecting,

accumulate text from documents containing the visually similar images, the text in a proximity of the visually similar images,

determine a name of a person from the accumulated text; and

an output component stored in the memory and operable on the processor to output the name of the person.

2. The system of claim 1, wherein the visually similar images are collected from the Internet.

3. The system of claim 1, wherein the analysis component is configured to compute a correlation between a name in the accumulated text and the face image.

4. The system of claim 1, wherein the output component is configured to output a database of person images, the person images annotated with information extracted from the accumulated text, the information including at least one of a name of the person, a birth date of the person, a gender of the person, and an occupation of the person.

5. The system of claim 4, wherein the system is configured to perform face recognition using a non-parametric face recognition engine based on instance search in the annotated database of person images.

6. One or more computer readable storage devices comprising computer executable instructions that, when executed by a computer processor, direct the computer processor to perform operations including:

receiving a query including an image;

automatically collecting at least one visually similar image to the included image and text from a file containing the visually similar image, the text being in a proximity of the visually similar image within the file;

determining a name of an entity in the included image based on the collecting; and

outputting the name of the entity.

7. The one or more computer readable storage devices of claim 6, wherein the image is a face image and the entity is a person.

8. The one or more computer readable storage devices of claim 7, further comprising filtering the accumulated text to obtain candidate names of the person.

9. The one or more computer readable storage devices of claim 7, further comprising training a binary classification model with a support vector machine (SVM), wherein the candidate name is determined based on a binary classifier.

10. The one or more computer readable storage devices of claim 6, further comprising detecting at least one visual feature from the image.

11. The one or more computer readable storage devices of claim 6, further comprising accumulating text from a proximity of the at least one visually similar image, wherein the name of the entity is determined based on the accumulated text.

12. The one or more computer readable storage devices of claim 6, further comprising computing a correlation between a name in the accumulated text and the included image.

13. The one or more computer readable storage devices of claim 6, further comprising performing multiple searches for visually similar images to the included image and aggregating accumulated text from searches that result in duplicate visually similar images.

14. The one or more computer readable storage media of claim 6, further comprising determining the name of the entity based on an algorithm comprising the equation:

v(t_q)=f({right arrow over (W)},{right arrow over (X)}_q)

wherein v is a probability that a candidate name t_q is the name of the entity in the included image, {right arrow over (W)} are model parameters that are learned from training data, and {right arrow over (X_q)} represents a feature vector corresponding to the candidate name t_q.

15. The one or more computer readable storage devices of claim 6, wherein the determining the name of the entity in the included image is based at least on a probability that a candidate name is the name of the entity in the included image equivalent to a function of a vector of the name of the entity in the included image, based on model parameters that are learned from training data, and a feature vector corresponding to the candidate name.

16. A computer implemented method of identifying a name of a person in an image, the method comprising:

receiving a query including a face image;

detecting at least one visual feature from the face image;

collecting at least one visually similar image to the face image, based on the detecting;

accumulating text from at least one document containing the at least one visually similar image, the text in a proximity of the at least one visually similar image;

determining a name of a person from the accumulated text; and

outputting the name of the person.

17. The method of claim 16, further comprising extracting terms and/or phrases from the accumulated text and filtering the terms or phrases with a predefined list.

18. The method of claim 16, further comprising outputting a database of person images, the person images annotated with information extracted from the accumulated text, the information including at least one of a name of the person, a birth date of the person, a gender of the person, and an occupation of the person.

19. The method of claim 16, further comprising determining the name Of the person based on the equation:

20. The method of claim 16 wherein the determining the name of the person represented by the face image from the accumulated text based on an optimization for each candidate name is based at least on a visual similarity between the face image and a candidate image and a correlation between each candidate name and candidate image.