Facial recognition with social network aiding

• US 10,031,927 B2
• Filed: 11/02/2015
• Issued: 07/24/2018
• Est. Priority Date: 08/07/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Petrou, David, Rabinovich, Andrew, Adam, Hartwig
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US14/929,958
• Publication Number:
  US 20160055182A1
• Time in Patent Office: 995 Days
• Field of Search: 382116, 707734
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A facial recognition search system identifies one or more likely names (or other personal identifiers) corresponding to the facial image(s) in a query as follows. After receiving the visual query with one or more facial images, the system identifies images that potentially match the respective facial image in accordance with visual similarity criteria. Then one or more persons associated with the potential images are identified. For each identified person, person-specific data comprising metrics of social connectivity to the requester are retrieved from a plurality of applications such as communications applications, social networking applications, calendar applications, and collaborative applications. An ordered list of persons is then generated by ranking the identified persons in accordance with at least metrics of visual similarity between the respective facial image and the potential image matches and with the social connection metrics. Finally, at least one person identifier from the list is sent to the requester.

Claims

1. A computer-implemented method, comprising:

receiving, from a client device, a visual query comprising one or more images;

identifying an object or object category associated with the one or more images of the visual query;

identifying visual similarity criteria associated with the identified object or object category associated with the one or more images of the visual query;

determining candidate image matches that are indicated as matching the one or more images of the visual query based on the visual similarity criteria associated with the object or object category;

in response to determining candidate image matches that are indicated as matching the one or more images of the visual query based on the visual similarity criteria, determining one or more measures of visual similarity between the respective one or more images of the visual query and each of the plurality of potential image matches;

ranking each of the candidate image matches in accordance with the one or more measures of visual similarity between the respective one or more images of the visual query and each of the candidate image matches, wherein ranking each of the candidate image matches is further based on social connection data from a social graph; and

providing at least one image from the ranking of candidate image matches to the client device as a related image based on the visual query.

2. The computer-implemented method of claim 1, comprising:

generating a related image interface comprising a visual identifier associated with the identified object or object category and a user-selectable link corresponding to the visual identifier,

wherein the user-selectable link corresponds to the at least one image from the ranking of candidate image matches provided to the client device as a related image based on the visual query.

3. The computer-implemented method of claim 2, comprising:

in response to receiving user input associated with the user-selectable link corresponding to the visual identifier, providing the at least one image from the ranking of candidate image matches for display on the client device.

4. The computer-implemented method of claim 1, wherein identifying an object or object category associated with the one or more images of the visual query comprises:

identifying text associated with the one or more images of the visual query; and

detecting the object or object category based on the identified text associated with the one or more images of the visual query.

5. The computer-implemented method of claim 1, wherein determining candidate image matches that are indicated as matching the one or more images of the visual query is further based on the social connection data from the social graph.

6. The computer-implemented method of claim 1, wherein the candidate image matches correspond to the identified object or object category associated with the one or more images of the visual query.

7. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

receiving, from a client device, a visual query comprising one or more images;

identifying an object or object category associated with the one or more images of the visual query;

identifying visual similarity criteria associated with the identified object or object category associated with the one or more images of the visual query;

determining candidate image matches that are indicated as matching the one or more images of the visual query based on the visual similarity criteria associated with the object or object category;

in response to determining candidate image matches that are indicated as matching the one or more images of the visual query based on the visual similarity criteria, determining one or more measures of visual similarity between the respective one or more images of the visual query and each of the plurality of potential image matches;

ranking each of the candidate image matches in accordance with the one or more measures of visual similarity between the respective one or more images of the visual query and each of the candidate image matches, wherein ranking each of the candidate image matches is further based on social connection data from a social graph; and

providing at least one image from the ranking of candidate image matches to the client device as a related image based on the visual query.

8. The system of claim 7, wherein the operations comprise:

generating a related image interface comprising a visual identifier associated with the identified object or object category and a user-selectable link corresponding to the visual identifier,

wherein the user-selectable link corresponds to the at least one image from the ranking of candidate image matches provided to the client device as a related image based on the visual query.

9. The system of claim 8, wherein the operations comprise:

in response to receiving user input associated with the user-selectable link corresponding to the visual identifier, providing the at least one image from the ranking of candidate image matches for display on the client device.

10. The system of claim 7, wherein identifying an object or object category associated with the one or more images of the visual query comprises:

identifying text associated with the one or more images of the visual query; and

detecting the object or object category based on the identified text associated with the one or more images of the visual query.

11. The system of claim 7, wherein determining candidate image matches that are indicated as matching the one or more images of the visual query is further based on the social connection data from the social graph.

12. The system of claim 7, wherein the candidate image matches correspond to the identified object or object category associated with the one or more images of the visual query.

13. A non-transitory computer-readable storage device storing instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving, from a client device, a visual query comprising one or more images;

identifying an object or object category associated with the one or more images of the visual query;

identifying visual similarity criteria associated with the identified object or object category associated with the one or more images of the visual query;

determining candidate image matches that are indicated as matching the one or more images of the visual query based on the visual similarity criteria associated with the object or object category;

in response to determining candidate image matches that are indicated as matching the one or more images of the visual query based on the visual similarity criteria, determining one or more measures of visual similarity between the respective one or more images of the visual query and each of the plurality of potential image matches;

ranking each of the candidate image matches in accordance with the one or more measures of visual similarity between the respective one or more images of the visual query and each of the candidate image matches, wherein ranking each of the candidate image matches is further based on social connection data from a social graph; and

providing at least one image from the ranking of candidate image matches to the client device as a related image based on the visual query.

14. The computer-readable storage device of claim 13, wherein the operations comprise:

generating a related image interface comprising a visual identifier associated with the identified object or object category and a user-selectable link corresponding to the visual identifier,

wherein the user-selectable link corresponds to the at least one image from the ranking of candidate image matches provided to the client device as a related image based on the visual query.

15. The computer-readable storage device of claim 14, wherein the operations comprise:

in response to receiving user input associated with the user-selectable link corresponding to the visual identifier, providing the at least one image from the ranking of candidate image matches for display on the client device.

16. The computer-readable storage device of claim 13, wherein identifying an object or object category associated with the one or more images of the visual query comprises:

identifying text associated with the one or more images of the visual query; and

detecting the object or object category based on the identified text associated with the one or more images of the visual query.

17. The computer-readable storage device of claim 13, wherein determining candidate image matches that are indicated as matching the one or more images of the visual query is further based on the social connection data from the social graph.