Computer-implemented method for performing similarity searches

• US 7,760,917 B2
• Filed: 07/12/2007
• Issued: 07/20/2010
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Shah, Munjal, Tengli, Ashwin, Gokturk, Salih Burak, Anguelov, Dragomir, Vanhoucke, Vincent, Lee, Kuang-chih
• Primary Examiner(s): KIM, CHONG R
• Application Number:
  US11/777,070
• Publication Number:
  US 20080080745A1
• Time in Patent Office: 1,104 Days
• Field of Search: None
• US Class Current: 382/115
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/179 metadata assisted face recognition

Abstract

A similarity search may be performed on the image of a person, using visual characteristics and information that is known about the person. The search identifies images of other persons that are similar in appearance to the person in the image.

Claims

1. A computer-implemented method for determining face similarity, the method comprising:

using one or more processors to perform steps comprising;

detecting a plurality of persons collectively depicted in a collection of images;

performing image analysis for individual persons that appear in one or more images in the collection, including determining a plurality of visual features and determining gender, age, or racial classifications for at least some of the persons depicted in the collection from performance of the image analysis, wherein the plurality of visual features comprise skin tone, hair color, hair length, or face type;

storing information determined from performing image analysis for the individual persons;

receiving one or more inputs from the user that (i) include an image of a person, and (ii) identify one or more gender, age, or racial classifications; and

performing a similarity search using said one or more inputs, in order to identify images of other persons depicted in one or more images of the collection that are both (i) similar in appearance to the person in the image input by the user, and (ii) match the gender, age, or racial classifications identified by the one or more inputs;

wherein performing the similarity search includes determining similarities between a plurality of visual features of the person depicted in the image input by the user and the plurality of visual features of persons depicted in the one or more images of the collection, wherein the determined similarities are weighted based upon the gender, age, or racial classifications identified by the one or more inputs.

2. The method of claim 1, wherein determining gender, age or racial classifications for at least some of the persons depicted in the collection includes determining gender, age, or racial information about the person in the image, at least in part, by performing analysis of text or metadata associated with the collection of images.

3. The method of claim 1, wherein performing the similarity search includes formulating a search query to include image data that represents one or more visual characteristics of a face of the person included in the one or more inputs.

4. The method of claim 2, wherein determining gender, age, or racial classifications for at least some of the persons further comprises identifying a label or a tag provided with the image.

5. The method of claim 2, wherein determining gender, age, or racial classifications for at least some of the persons further comprises identifying a text contained as part of the image.

6. A system for determining face similarity, the system comprising:

an analysis module that is configured to perform image analysis to identify, for individual images in a collection of images, (i) a face from the image, and (ii) at least one of text or metadata associated with the face of the image;

a characteristic determination module that is configured to identify, for individual images in the collection, (i) a plurality of visual features, and (ii) one or more gender, age, or racial classifications based at least in part on image analysis performed by the analysis module;

a data store to store information determined from the image analysis module and the characteristic determination module;

a comparison module that identifies a query comprising (i) an image input of a face of a given person, the image input being received from a user, and (ii) a gender, age, or racial classification as determined from the image input or from other input of the user;

wherein the comparison module performs a similarity search by comparing the query with information in the data store in order to identify one or more faces of persons other than the given person in the collection of images that are determined as being both (i) similar to the face of the query image input, and (ii) of a same gender, age, or racial classification of the query;

and wherein the similarity search includes determining similarities between a plurality of visual features of the face of the query image input and the plurality of visual features of persons depicted in one or more images of the collection, wherein the determined similarities are weighted based upon the gender, age, or racial classification for the face of the query image input.

7. The system of claim 6, wherein the characteristic determination module is configured to determine any of the one or more gender, age, or racial classifications using a combination of the text and metadata and any image data representing the face of the image.

8. The system of claim 6, wherein the characteristic determination module identifies at least one of text or metadata associated with the image by identifying a text contained in a file that includes the image of the person.

9. The system of claim 6, wherein the characteristic determination module identifies at least one of text or metadata associated with the image by identifying a label or a tag provided with the image.