Face recognition using social data

• US 8,341,145 B2
• Filed: 12/20/2010
• Issued: 12/25/2012
• Est. Priority Date: 12/20/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dodson, Bradley, Nichols, David, Webster, Chung, Scaria, Joel
• Primary Examiner(s): Lee, Wilson
• Assistant Examiner(s): Le, Jessica N
• Application Number:
  US12/973,881
• Publication Number:
  US 20120158700A1
• Time in Patent Office: 736 Days
• Field of Search: None
• US Class Current: 707/722
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

Faces may be indexed and identified using visual and social criteria. In one example, the visual features of a face are quantified, and the quantification of the features is represented in a vector. Aspects of the vector are then represented in the form of text strings. Social context surrounding the face is also represented in the form of text strings. The text strings—both the visual-based strings and/or the social-based strings—are associated with the face, and are stored in an index. The association of these strings with the face then may make the face text-searchable on both its visual and social features. Searches on these visual and/or social features may be used to help to assist in identifying new images of faces, or to propose tags for users to apply to photos.

Claims

1. One or more computer-readable storage media that store executable instructions to index faces based on visual and social data, wherein the executable instructions, when executed by a computer, cause the computer to perform acts comprising:

analyzing an image of a first face to produce a vector that quantifies features of said first face;

creating one or more first text strings that represent said vector, said creating of said one or more text strings comprising;

defining a first set of planes in a space in which said vector exists;

determining which side of each of said planes said vector lies;

encoding which side of each of said planes said vector lies in a text string, said text string being one of said one or more first text strings;

defining a plurality of sets of planes, said first set of planes being one of said plurality of sets of planes; and

creating a separate text string based on side-of-plane decisions for each of said plurality of sets of planes;

obtaining social data concerning a first photo in which said first face appears;

associating said first face with said one or more first text strings and said social data;

storing said first face in a database in association with said one or more first text strings and said social data;

receiving an image of a second face;

creating one or more second text strings based on a visual appearance of said second face;

obtaining social data from a second photo in which said second face appears;

creating a text query comprising said one or more second text strings and said social data;

obtaining, based on said query, a set of results that comprises said first face, wherein including of said first face in said results is based on said first face being indexed by terms that are in said query; and

performing an action based on said results, said action comprising providing, to a user, one or more faces that are of the same person as said second face, said first face being determined to be of the same person as said second face based on said first face'"'"'s inclusion in said results or based on said first face'"'"'s rank in said results.

2. The one or more computer-readable storage media of claim 1, wherein said creating of said one or more first text strings comprises:

using a locality sensitive hash that generated text strings such that similarity of text strings generated for vectors is positively related to proximity of the vectors.

3. The one or more computer-readable storage media of claim 1, wherein said social data comprises who has uploaded said first photo, wherein said first face is indexed in said database by said one or more first text strings and by a name of a person who has uploaded said first photo.

4. The one or more computer-readable storage media of claim 1, wherein said social data is based on who has tagged said first photo, wherein said first face is indexed in said database by said one or more first text strings and by a name of a person who has tagged said first photo.

5. The one or more computer-readable storage media of claim 1, wherein said social data is based on who appears in said first photo with said first face, wherein said first face is indexed in said database by said one or more first text strings and by a name of a person who appears in said first photo.

6. The one or more computer-readable storage media of claim 1, wherein said social data is based on friend relationships in a social network to which said first photo has been uploaded.

7. The one or more computer-readable storage media of claim 1, wherein said action comprises:

proposing, to said user, one or more tags for said second face.

8. A method of indexing and identifying faces based on visual and social data, the method comprising:

extracting, using a processor, an image of a first face from a first photo;

analyzing, using the processor, said first face to produce a vector that quantifies features of said first face;

creating, using the processor, one or more first text strings based on said vector, said creating of said one or more text strings comprising;

defining a first set of planes in a space in which said vector exists;

determining which side of each of said planes said vector lies;

encoding which side of each of said planes said vector lies in a text string, said text string being one of said one or more first text strings;

defining a plurality of sets of planes, said first set of planes being one of said plurality of sets of planes; and

creating a separate text string based on side-of-plane decisions for each of said plurality of sets of planes;

obtaining social data concerning a first photo in which said first face appears;

storing, in a database, an association between said one or more first text strings and said social data;

receiving an image of a second face;

creating one or more second text strings based on a visual appearance of said second face;

creating a query that comprises said one or more second text strings and social data derived from a second photo in which said second face appears;

receiving results based on said query, said results comprising said first face based on said first face'"'"'s being associated with text in said query and being indexed by terms that are in said query; and

performing, using the processor, an action based on said results, said action comprising providing, to a user, one or more faces that are of the same person as said second face, said first face being determined to be of the same person as said second face based on said first face'"'"'s inclusion in said results or based on said first face'"'"'s rank in said results.

9. The method of claim 8, wherein said creating of said one or more first text strings comprises using a locality-sensitive hash of said vector.

10. The method of claim 8, wherein said social data comprises an identity of a person who uploaded said first photo.

11. The method of claim 8, wherein said social data comprises an identity of a person who appears, with said first face, in said first photo.

12. The method of claim 8, wherein said social data comprises an identity of a person who has tagged said first photo.

13. A system for identifying a first face based on visual and social data, the system comprising:

a memory;

a processor;

a database that stores faces indexed by text strings that represent visual characteristics of said faces and also by social data relating to said faces; and

a face processing component that receives an image of said first face and of a second face, that creates a vector that represents a visual appearance of said first face, that creates one or more text strings that represent said vector, that creates one or more text strings based on a visual appearance of said second face, that obtains social data relating to a photo in which said first face appears, that forms a text query comprising said one or more text strings and said social data, that obtains, from said database, results based on said query, said results comprising said first face, said first face being included in said results based on said first base being indexed by terms that are in said query, said fact processing component taking an action based on said results,

wherein said results comprise said second face, said action comprising providing, to a user, one or more faces that are of the same person as said second face, said first face being determined to be of the same person as said second face based on said first face'"'"'s inclusion in said results or based on said first face'"'"'s rank in said results, and

wherein said text strings are created by defining a first set of planes in a space in which said vector exists, determining which side of each of said planes said vector lies, encoding which side of each of said planes said vector lies in a text string, defining a plurality of sets of planes in which said first set of planes are one of said plurality of sets of planes, and creating a separate text string based on side-of-plane decisions for each of said plurality of sets of planes.

14. The system of claim 13, wherein said action comprises proposing an identity associated with said second face as a tag for said first face.