Facial Recognition with Social Network Aiding

• US 20110038512A1
• Filed: 08/05/2010
• Published: 02/17/2011
• Est. Priority Date: 08/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Rabinovich, Andrew, Adam, Hartwig, Petrou, David
• Granted Patent: US 8,670,597 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A facial recognition search system identifies one or more likely names (or other personal identifiers) corresponding to the facial image(s) in a query as follows. After receiving the visual query with one or more facial images, the system identifies images that potentially match the respective facial image in accordance with visual similarity criteria. Then one or more persons associated with the potential images are identified. For each identified person, person-specific data comprising metrics of social connectivity to the requester are retrieved from a plurality of applications such as communications applications, social networking applications, calendar applications, and collaborative applications. An ordered list of persons is then generated by ranking the identified persons in accordance with at least metrics of visual similarity between the respective facial image and the potential image matches and with the social connection metrics. Finally, at least one person identifier from the list is sent to the requester.

Claims

1. A computer-implemented method of processing a visual query including a facial image performed on a server system having one or more processors and memory storing one or more programs for execution by the one or more processors to perform the method, comprising:

receiving from a requester, a visual query comprising one or more facial images including a respective facial image;

identifying potential image matches that potentially match the respective facial image in accordance with visual similarity criteria, the potential image matches comprising images from one or more image sources identified in accordance with data regarding the requester;

identifying one or more persons associated with the potential image matches;

retrieving, for each identified person, person-specific data comprising social connection metrics of social connectivity to the requester obtained from a plurality of applications selected from the group consisting of communication applications, social networking applications, calendar applications, and collaborative applications;

generating an ordered list of persons by ranking the one or more identified persons in accordance with one or more metrics of visual similarity between the respective facial image and the potential image matches and in accordance with ranking information comprising at least the social connection metrics; and

sending to the requester at least one person identifier from the ordered list.

2. The computer-implemented method of claim 1,

wherein the plurality of applications include one or more communication applications selected from the group consisting of:

social networking applications, social micro-blogs, email applications, and instant messaging applications;

wherein the method further comprises;

determining an amount of communication between a respective person and the requester on the one or more communication applications; and

determining a ranking score for the respective person, wherein a factor in determining the ranking score for the respective person is the determined amount of communication between the respective person and the requester on the one or more communication applications.

3. The computer-implemented method of claim 2, including:

determining whether the amount of communication between the respective person and the requester on the one or more communication applications exceeds a threshold;

wherein a factor in determining the ranking score for the respective person is the determination of whether the amount of communication between the respective person and the requester on the one or more communication applications exceeds the threshold.

4. The computer implemented method of claim 1, further comprising:

determining whether the requester and a respective person are directly connected in a respective social networking application, and

determining a ranking score for the respective person, wherein a factor in determining the ranking score for the respective person is the determination of whether the requester and the respective person are directly connected in a respective social networking application.

5. The computer-implemented method of claim 1,

wherein the person-specific data for a respective person further comprises characteristics derived from other images of the respective person;

and

wherein the ranking is further in accordance with similarity between the received query and the characteristics derived from other images of the respective person.

6. The computer-implemented method of claim 5,

wherein the characteristics derived from other images of the respective person include image capture date and time information;

and

wherein the method further comprises;

determining one more similarity metrics in accordance with an extent to which the received query has image capture date and time information similar to the date and time information of the other images of the respective person; and

determining a ranking score for the respective person, wherein one or more factors in determining the ranking score for the respective person are the one or more similarity metrics.

7. The computer-implemented method of claim 5,

wherein the characteristics derived from other images of the respective person include occurrences information regarding an amount of occurrences of the respective person in images from the one or more image sources;

and

wherein a factor in determining the ranking score for the respective person is the occurrences information for the respective person.

8. The computer-implemented method of claim 5,

wherein the characteristics derived from other images of the respective person comprises visual factors including one or more of:

an indoor habitat factor, an outdoor habitat factor, a gender factor, a race factor, a glasses factor, a facial hair factor, a head hair factor, a headwear factor, clothing factor, and an eye color factor; and

wherein factors in determining the ranking score for the respective person include the visual factors for the respective person.

9. The computer-implemented method of claim 1,

wherein the visual query comprises a plurality of facial images including at least the respective facial image and a second facial image;

wherein the method further comprises;

identifying potential second image matches that potentially match the second facial image in accordance with visual similarity criteria, the second potential image matches comprising images from one or more image sources identified in accordance with data regarding the requester;

identifying a second person associated with the second facial image matches;

retrieving, for each identified person, person-specific data comprising second social connection metrics of social connectivity to the second person obtained from the plurality of applications; and

generating the ordered list of persons by ranking the one or more identified persons further in accordance with ranking information comprising at least the second social connection metrics.

10. The computer-implemented method of claim 9,

wherein the person-specific data for a respective person further comprises characteristics derived from other images of the respective person, the characteristics derived from other images of the respective person including information regarding an amount of co-occurrences of the respective person and the second person in images from the one or more image sources;

and

wherein the method further comprises;

determining a ranking score for the respective person, wherein a factor in determining the ranking score for the respective person is the amount of co-occurrences of the person and the second person in images from the one or more image sources.

11. The computer-implemented method of claim 1, further comprising:

prior to sending the requester at least one person identifier, checking an opt-in list and determining that the person identifier is releasable to the requester.

12. The computer-implemented method of claim 1,

wherein for a respective identified person, the person-specific data further comprises one or more of:

name, address, occupation, group memberships, interests, age, hometown, personal statistics, and work information for the respective person; and

wherein the method further comprises;

retrieving person-specific data for the requester that also includes one or more of;

name, address, occupation, group memberships, interests, age, hometown, personal statistics, and work information for the respective person;

determining one or more personal similarity metrics in accordance with an extent to which the person-specific data of the requester is similar to the person-specific data of the respective identified person; and

determining a ranking score for the respective identified person, wherein one or more factors in determining the ranking score for the respective identified person are the one or more personal similarity metrics.

13. The computer-implemented method of claim 1, further comprising:

obtaining current location information for the requester;

obtaining current location information for a respective identified person; and

determining a ranking score for the respective identified person, wherein a factor in determining the ranking score for the respective identified person is whether current location information for the requester matches the current location information for the respective identified person.

14. The computer-implemented method of claim 1, further comprising:

after the sending, receiving from the requester a selection of a person identifier; and

storing the respective facial image of the visual query as an additional image of a respective person corresponding to the selected person identifier.

15. The computer-implemented method of claim 1,

wherein the visual query comprises a plurality of facial images including at least the respective facial image and a second facial image;

and

wherein the method further comprises;

prior to identifying potential image matches, receiving a selection of the respective facial image from the requester.

16. A server system, for processing a visual query including a facial image, comprising:

one or more processors for executing programs;

memory storing one or more programs be executed by the one or more processors;

the one or more programs comprising instructions for;

receiving from a requester, a visual query comprising one or more facial images including a respective facial image;

identifying potential image matches that potentially match the respective facial image in accordance with visual similarity criteria, the potential image matches comprising images from one or more image sources identified in accordance with data regarding the requester;

identifying one or more persons associated with the potential image matches;

retrieving, for each identified person, person-specific data comprising social connection metrics of social connectivity to the requester obtained from a plurality of applications selected from the group consisting of communication applications, social networking applications, calendar applications, and collaborative applications;

generating an ordered list of persons by ranking the one or more identified persons in accordance with one more metrics of visual similarity between the respective facial image and the potential image matches and in accordance with ranking information comprising at least the social connection metrics; and

sending to the requester at least one person identifier from the ordered list.

17. The server system of claim 16,

wherein the plurality of applications include one or more communication applications selected from the group consisting of:

social networking applications, social micro-blogs, email applications, and instant messaging applications;

wherein the method further comprises;

determining an amount of communication between a respective person and the requester on the one or more communication applications; and

determining a ranking score for the respective person, wherein a factor in determining the ranking score for the respective person is the determined amount of communication between the respective person and the requester on the one or more communication applications.

18. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving from a requester, a visual query comprising one or more facial images including a respective facial image;

identifying potential image matches that potentially match the respective facial image in accordance with visual similarity criteria, the potential image matches comprising images from one or more image sources identified in accordance with data regarding the requester;

identifying one or more persons associated with the potential image matches;

retrieving, for each identified person, person-specific data comprising social connection metrics of social connectivity to the requester obtained from a plurality of applications selected from the group consisting of communication applications, social networking applications, calendar applications, and collaborative applications;

generating an ordered list of persons by ranking the one or more identified persons in accordance with one more metrics of visual similarity between the respective facial image and the potential image matches and in accordance with ranking information comprising at least the social connection metrics; and

sending to the requester at least one person identifier from the ordered list.

19. The computer readable storage medium of claim 18,

wherein the plurality of applications include one or more communication applications selected from the group consisting of:

social networking applications, social micro-blogs, email applications, and instant messaging applications;

wherein the method further comprises;

determining an amount of communication between a respective person and the requester on the one or more communication applications; and

determining a ranking score for the respective person, wherein a factor in determining the ranking score for the respective person is the determined amount of communication between the respective person and the requester on the one or more communication applications.